CN115550944B - Dynamic service placement method based on edge calculation and deep reinforcement learning in Internet of vehicles - Google Patents

Abstract

The invention discloses a dynamic service placement method based on edge calculation and deep reinforcement learning in the Internet of vehicles, which comprises the following steps: 1) Establishing a network and service request model, and acquiring information related to the network and service request; 2) Establishing a network and service request calculation model; 3) Constructing a state space, an action space, a strategy function and a reward function; 4) Constructing an actor network and a criticism network, and training the actor network and the criticism network; 5) Generating a service placement strategy by the actor network and inputting the strategy into the criticizing home network; 6) And (3) the criticizing home network evaluates the strategy quality of the service placement strategy, if the evaluation is not passed, updating actor network parameters, returning to the step (5), and if the evaluation is passed, outputting the service placement strategy. The present invention minimizes maximum edge resource usage and service delays while taking into account vehicle mobility, changing demands, and dynamics of different types of service requests.

Description

A dynamic service placement method based on edge computing and deep reinforcement learning in Internet of Vehicles

Technical Field

The present invention relates to the field of Internet of Vehicles, and specifically to a dynamic service placement method based on edge computing and deep reinforcement learning in the Internet of Vehicles.

Background Art

The Internet of Vehicles is an interactive network consisting of information such as vehicle location, speed, and route. The rapid development of communication technology has brought many new possibilities to the current field of Internet of Vehicles. Among them, the emergence of the fifth generation of mobile communication technology has made the Internet of Vehicles more intelligent and the service coverage has been further expanded. However, as delay-sensitive applications such as intelligent voice assistants and autonomous driving have become the most popular applications in the field of Internet of Vehicles, the traditional cloud computing paradigm has gradually failed to meet user needs. The European Telecommunications Standards Institute introduced mobile edge computing into the field of Internet of Vehicles, expanding the storage and computing resources of cloud computing, bringing them closer to users, and meeting users' requirements for high reliability, low latency, and security of intelligent applications.

In the Internet of Vehicles, vehicles communicate with the infrastructure to obtain services such as media downloads, cooperative messages, and decentralized environmental notification messages, thereby achieving coordination in applications such as remote driving, parking space discovery, and navigation. In the edge computing paradigm, multiple services can be deployed on edge servers to fully utilize computing resources and storage resources. Service placement is one of the research hotspots in the field of Internet of Vehicles. Specifically, service placement is the mapping of services to edge servers in the Internet of Vehicles to meet the needs of the requested services while efficiently using edge resources. From the user's perspective, it is very important to minimize the latency of vehicle-aware services. From the service provider's perspective, it is necessary to maximize the utilization of edge resources while keeping the resource load balance between servers as much as possible.

Summary of the invention

The purpose of the present invention is to provide a dynamic service placement method based on edge computing and deep reinforcement learning in a vehicle network, comprising the following steps:

1) Establish a network and service request model to obtain network and service request related information;

The network and service request related information includes edge server information, vehicle information, and service information;

The edge server information includes an edge server set E, an edge server e, and a remaining resource capacity C _e of the edge server e;

The vehicle information includes a vehicle set V.

The service information includes the service set S, the number of vehicles λ _s requesting service s, the number of vehicles ε that can process one service instance at a time (such as media file downloading, cooperative awareness messages and environmental notification services in the Internet of Vehicles environment) or provide parallel connections, the time t and vehicle location loc specified in the service request message, the amount of resources R _s consumed by the edge server deploying service s, and the delay requirement threshold D _s .

2) Establish a network and service request calculation model;

The network and service request calculation model includes a total service delay calculation model and an edge resource utilization calculation model;

The total service delay calculation model is as follows:

In the formula, is the total service delay; are the propagation delay and queuing delay; dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium;

When the number of vehicles requesting service s is _λs ≤ε, the queuing delay is When the number of vehicles requesting service s λ _s ＞ ε, the queuing delay Satisfy the following formula:

In the formula, the quantity difference λ′ _s =λ _s -ε;

Propagation Delay As shown below:

Where dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium.

The edge resource utilization calculation model is as follows:

Edge resource utilization is the ratio between the resources consumed by the service instance and the available resources of the edge server, as follows:

In the formula, the parameters C _e is the remaining resource capacity of edge server e; is the edge resource utilization rate; _Rs is the amount of resources consumed by the edge server to deploy service s.

3) Construct state space, action space, strategy function and reward function;

The state space is characterized by the state space set ω, namely:

ω={[v ₁ ,loc ₁ ,s],[v ₂ ,loc ₂ ,s],...,[v _n ,loc _n ,s]} _t (6)

Where s∈S; v ₁ ,v ₂ ,...,v _n is a set of vehicles; loc ₁ ,loc ₂ ,...,loc _n is the set of vehicle locations requesting service s at time t.

The action space is used to describe the actions taken when placing a service on an edge server;

The action a taken at a given time t is as follows:

Where π is the policy function required to generate actions for the observation set ω in time unit t; Indicates that service s is deployed on edge server e; Indicates that service s is not deployed on edge server e.

The policy function π is a function executed by the actor network to map the state space to the action space, i.e., π:ω→a;

The goal of the policy function π is to minimize the maximum edge resource usage and service delay, and to control the relative importance of resource usage and service delay by using the parameter β. The policy function π is expressed as

In the formula, β is the weight coefficient;

The constraints of the policy function π include the mapping constraints Delay Constraint Resource Constraints

The reward function is as follows:

In the formula, is the immediate reward. γ is the reward coefficient. is the service delay at time t;

4) Constructing the actor network and the critic network, and training the actor network and the critic network;

The loss function during the training of the critic network As shown below:

Where θ is the critic network parameter; is the target value used to evaluate the quality of the strategy; _Qi (ω, a; θ) is the strategy quality of the service placement strategy; is the number of available resource units in the edge server;

5) The actor network generates service placement strategies and inputs them into the critic network;

6) The critic network evaluates the policy quality of the service placement strategy. If the evaluation fails, the actor network parameters are updated and the process returns to step 5. If the evaluation passes, the service placement strategy is output.

The method for evaluating the strategy quality of the service placement strategy by the critic network includes: determining the critic network loss function Whether it converges. If so, the evaluation passes. Otherwise, the evaluation fails.

It is worth noting that the present invention proposes a three-layer Internet of Vehicles architecture based on edge computing and takes into account the problem of dynamic service placement. The optimization goal is to minimize the maximum edge resource usage (from the perspective of the service provider) and service delay (from the perspective of the user).

In addition, this paper proposes a service placement framework based on deep reinforcement learning, which consists of a policy function (actor network) and a value function (critic network). The actor network makes a service placement strategy, while the critic network evaluates the performance of the decision made by the actor network based on the delay observed by the vehicle.

The technical effect of the present invention is unquestionable. The present invention provides a dynamic service placement method based on edge computing and deep reinforcement learning in the Internet of Vehicles. The method proposes a dynamic service placement framework based on deep reinforcement learning in the Internet of Vehicles, which aims to minimize the maximum edge resource usage and service delay while considering the mobility of vehicles, changing needs and the dynamics of requests for different types of services.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 shows a three-layered Internet of Vehicles architecture based on edge computing;

Figure 2 shows the structure of the intelligent agent;

FIG3 is a flow chart of the present invention.

DETAILED DESCRIPTION

The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

Embodiment 1:

Referring to FIG. 1 to FIG. 3 , a dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles includes the following steps:

1) Establish a network and service request model to obtain network and service request related information;

The network and service request related information includes edge server information, vehicle information, and service information;

The edge server information includes an edge server set E, an edge server e, and a remaining resource capacity C _e of the edge server e;

The vehicle information includes a vehicle set V.

The service information includes the service set S, the number of vehicles λ _s requesting service s, the number of vehicles ε that can process one service instance at a time (such as media file downloading, cooperative awareness messages and environmental notification services in the Internet of Vehicles environment) or provide parallel connections, the time t and vehicle location loc specified in the service request message, the amount of resources R _s consumed by the edge server deploying service s, and the delay requirement threshold D _s .

2) Establish a network and service request calculation model;

The network and service request calculation model includes a total service delay calculation model and an edge resource utilization calculation model;

The total service delay calculation model is as follows:

In the formula, is the total service delay; are the propagation delay and queuing delay; dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium;

When the number of vehicles requesting service s is _λs ≤ε, the queuing delay is When the number of vehicles requesting service s λ _s ＞ ε, the queuing delay Satisfy the following formula:

In the formula, the quantity difference λ′ _s =λ _s -ε;

Propagation Delay As shown below:

Where dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium.

The edge resource utilization calculation model is as follows:

Edge resource utilization is the ratio between the resources consumed by the service instance and the available resources of the edge server, as follows:

In the formula, the parameters C _e is the remaining resource capacity of edge server e; is the edge resource utilization rate; _Rs is the amount of resources consumed by the edge server to deploy service s.

3) Construct state space, action space, strategy function and reward function;

The state space is characterized by the state space set ω, namely:

ω={[v ₁ ,loc ₁ ,s],[v ₂ ,loc ₂ ,s],...,[v _n ,loc _n ,s]} _t (6)

Where s∈S; v ₁ ,v ₂ ,...,v _n is a set of vehicles; loc ₁ ,loc ₂ ,...,loc _n is the set of vehicle locations requesting service s at time t.

The action space is used to describe the actions taken when placing a service on an edge server;

The action a taken at a given time t is as follows:

Where π is the policy function required to generate actions for the observation set ω in time unit t; Indicates that service s is deployed on edge server e; Indicates that service s is not deployed on edge server e.

The policy function π is a function executed by the actor network to map the state space to the action space, i.e., π:ω→a;

The goal of the policy function π is to minimize the maximum edge resource usage and service delay, and to control the relative importance of resource usage and service delay by using the parameter β. The policy function π is expressed as

In the formula, β is the weight coefficient;

The principle of the policy function π is: iterate the service set and the edge server set through the subscripts s and e to find the maximum edge resource usage and service delay, and then minimize them to obtain the corresponding policy function π.

The constraints of the policy function π include the mapping constraints Delay Constraint Resource Constraints

The reward function is as follows:

In the formula, is the immediate reward. γ is the reward coefficient. is the service delay at time t;

4) Constructing the actor network and the critic network, and training the actor network and the critic network;

The loss function during the training of the critic network As shown below:

Where θ is the critic network parameter; is the target value used to evaluate the quality of the strategy; _Qi (ω, a; θ) is the strategy quality of the service placement strategy; is the number of available resource units in the edge server;

5) The actor network generates service placement strategies and inputs them into the critic network;

6) The critic network evaluates the policy quality of the service placement strategy. If the evaluation fails, the actor network parameters are updated and the process returns to step 5. If the evaluation passes, the service placement strategy is output.

The method for evaluating the strategy quality of the service placement strategy by the critic network includes: determining the critic network loss function Whether it converges. If so, the evaluation passes. Otherwise, the evaluation fails.

Embodiment 2:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles includes the following steps:

1) Establish a network and service request model to obtain edge server information, vehicle information, and service information.

The server information, vehicle information and service information include the edge server set E, edge server e, the remaining resource capacity C _e of edge server e, the vehicle set V and the service set S, the number of vehicles λ _s requesting service s, the number of vehicles ε that can process one service instance at a time (such as media file downloading, cooperative awareness messages and environmental notification services in the Internet of Vehicles environment) or can provide parallel connections, the time t and vehicle location loc specified in the service request message, the amount of resources R _s consumed by the edge server to deploy service s, and the delay requirement threshold D _s .

2) Establish a computational model.

2.1) Total service delay modeling. The entire edge vehicle network system is modeled as an M/D/1 queue. When requesting service s from edge server e, the total service delay of the vehicle is It refers to the total time from when the vehicle sends a service request to when the edge server receives the corresponding response. Total service delay Propagation delay and queuing delay composition:

If λ _s ≤ ε, the queuing delay is 0. If λ _s ＞ε, a queue is created and the average queuing delay of service s on the edge server will be as follows:

Where λ′ _s = λ _s - ε, and the average propagation delay is calculated as the ratio of distance to the propagation speed in the medium, as follows:

Where dist(v,s) is the Euclidean distance between vehicle v and the edge server where service s is deployed, and c is the propagation speed of the signal through the communication medium. Therefore, the total service delay is as follows:

2.2) Edge resource utilization modeling. Edge resource utilization is the ratio between the resources consumed by the service instance and the available resources of the edge server, as follows:

in,

3) Design state space. At a given time t, the state space set describes the network environment. The agent observes the environment to form a state space set ω from the service request model, as shown below:

ω={[v ₁ ,loc ₁ ,s],[v ₂ ,loc ₂ ,s],...,[v _n ,loc _n ,s]} _t . (6)

Where s∈S, v ₁ ,v ₂ ,...,v _n is a set of vehicles, and loc ₁ ,loc ₂ ,...,loc _n is the set of vehicle locations requesting service s at time t.

4) Designed action space. The action space describes the actions taken by the policy module when placing services on edge servers. The actions taken at a given time t are as follows:

where π is the policy function required to generate actions for a set of observations ω at time unit t, and the binary variable Given a matrix indicating the location of service s on edge server e, Indicates that service s is deployed on edge server e. Otherwise, Indicates that service s is not deployed on edge server e.

5) Design the policy function. The policy function π is a function executed by the actor network to map the state space to the action space π:ω→a. The goal of the policy function π is to minimize the maximum edge resource usage and service delay, and to control the relative importance of resource usage and service delay by using the parameter β. The policy function π is expressed as

The policy function is also subject to mapping constraints, Delay constraints, and resource constraints,

6) Design a reward function. At each time unit t, in response to the actions taken by the agent's actor network, the system receives an immediate reward from the environment As shown below:

7) Build a critic network, which is responsible for evaluating the decision quality Q(ω,a) made by the actor network. Input the above states, actions and rewards to train the critic network, which updates its parameters θ to minimize the loss function As shown below:

Where y _t is the target value. The replay memory M is further used to store the experience of training the critic network. The critic network uses the replay memory to obtain experience after a random period of time and optimize the network parameters to obtain better performance.

8) After the training of the actor network and the critic network converges through the above steps, the actor network is able to find the best placement strategy for services while considering the mobility and dynamics of vehicles in different types of service requests. The critic network can evaluate the quality of the actor network's strategy through the value function.

Embodiment 3:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles includes the following steps:

1) Establish the network and service request model and obtain network and service request related information.

2) Establish a network and service request calculation model;

3) Construct state space, action space, strategy function and reward function;

4) Constructing the actor network and the critic network, and training the actor network and the critic network;

5) The actor network generates service placement strategies and inputs them into the critic network;

6) The critic network evaluates the policy quality of the service placement strategy. If the evaluation fails, the actor network parameters are updated and the process returns to step 5. If the evaluation passes, the service placement strategy is output.

Embodiment 4:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles, the main content of which is shown in Example 3, wherein the network and service request related information include edge server information, vehicle information, and service information;

The edge server information includes an edge server set E, an edge server e, and a remaining resource capacity C _e of the edge server e;

The vehicle information includes a vehicle set V.

The service information includes the service set S, the number of vehicles λ _s requesting service s, the number of vehicles ε that can process one service instance at a time or provide parallel connections, the time t and vehicle location loc specified in the service request message, the amount of resources R _s consumed by the edge server deploying service s, and the delay requirement threshold D _s ; the service instances include media file downloading, cooperative awareness messaging, and environmental notification services in the Internet of Vehicles environment.

Embodiment 5:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles, the main content of which is shown in Example 3, wherein the network and service request calculation model includes a total service delay calculation model and an edge resource utilization rate calculation model;

The total service delay calculation model is as follows:

In the formula, is the total service delay; are the propagation delay and queuing delay; dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium;

When the number of vehicles requesting service s is _λs ≤ε, the queuing delay is When the number of vehicles requesting service s λ _s ＞ ε, the queuing delay Satisfy the following formula:

In the formula, the quantity difference λ′ _s =λ _s -ε;

Propagation Delay As shown below:

Where dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium.

The edge resource utilization calculation model is as follows:

Edge resource utilization is the ratio between the resources consumed by the service instance and the available resources of the edge server, as follows:

In the formula, the parameters C _e is the remaining resource capacity of edge server e; is the edge resource utilization rate; _Rs is the amount of resources consumed by the edge server to deploy service s.

Embodiment 6:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles, the main content of which is shown in Example 3, wherein the state space is represented by a state space set ω, that is:

ω={[v ₁ ,loc ₁ ,s],[v ₂ ,loc ₂ ,s],...,[v _n ,loc _n ,s]} _t (6)

Where s∈S; v ₁ ,v ₂ ,...,v _n is a set of vehicles; loc ₁ ,loc ₂ ,...,loc _n is the set of vehicle locations requesting service s at time t.

Embodiment 7:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles, the main content of which is shown in Example 3, wherein the action space is used to describe the actions taken when placing a service on an edge server;

The action a taken at a given time t is as follows:

Where π is the policy function required to generate actions for the observation set ω in time unit t; Indicates that service s is deployed on edge server e; Indicates that service s is not deployed on edge server e.

Embodiment 8:

A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles, the main content of which is shown in Example 3, wherein the policy function π is a function executed by an actor network, which is used to map the state space to the action space, that is, π:ω→a;

The goal of the policy function π is to minimize the maximum edge resource usage and service delay, and to control the relative importance of resource usage and service delay by using the parameter β;

The policy function π is expressed as follows:

In the formula, β is the weight coefficient.

The constraints of the policy function π include the mapping constraints Delay Constraint Resource Constraints

Embodiment 9:

A dynamic service placement method based on edge computing and deep reinforcement learning in a vehicle network, the main content of which is shown in Example 3, wherein the reward function is as follows:

In the formula, is the immediate reward. γ is the reward coefficient. is the service delay at time t;

Embodiment 10:

A dynamic service placement method based on edge computing and deep reinforcement learning in a vehicle network, the main content of which is shown in Example 3, wherein the loss function in the critic network training process is As shown below:

Where θ is the critic network parameter; is the target value used to evaluate the quality of the strategy; _Qi (ω, a; θ) is the strategy quality of the service placement strategy; is the number of available resource units in the edge server;

Embodiment 11:

A dynamic service placement method based on edge computing and deep reinforcement learning in a vehicle network, the main content of which is shown in Example 3, wherein the method for evaluating the policy quality of the service placement strategy by the critic network includes: determining the critic network loss function Whether it converges. If so, the evaluation passes. Otherwise, the evaluation fails.

Claims (5)

1. A dynamic service placement method based on edge computing and deep reinforcement learning in an Internet of Vehicles, characterized in that it includes the following steps: 1) Establish a network and service request model to obtain network and service request related information; 2) Establish a network and service request calculation model; 3) Construct state space, action space, strategy function and reward function; 4) Constructing the actor network and the critic network, and training the actor network and the critic network; 5) The actor network generates service placement strategies and inputs them into the critic network; 6) The critic network evaluates the policy quality of the service placement strategy. If the evaluation fails, the actor network parameters are updated and the process returns to step 5. If the evaluation passes, the service placement strategy is output. The network and service request calculation model includes a total service delay calculation model and an edge resource utilization calculation model; The total service delay calculation model is as follows: In the formula, is the total service delay; are the propagation delay and queuing delay; dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium; When the number of vehicles requesting service s is _λs ≤ε, the queuing delay is When the number of vehicles requesting service s λ _s ＞ ε, the queuing delay Satisfy the following formula: In the formula, the quantity difference λ′ _s =λ _s -ε; Propagation Delay As shown below: Where dist(v,s) is the Euclidean distance between vehicle v and the edge server deployed by service s; c is the propagation speed of the signal through the communication medium; The edge resource utilization calculation model is as follows: Edge resource utilization is the ratio between the resources consumed by the service instance and the available resources of the edge server, as follows: In the formula, the parameters C _e is the remaining resource capacity of edge server e; is the edge resource utilization rate; _Rs is the amount of resources consumed by deploying service s on the edge server; The state space is characterized by the state space set ω, namely: ω={[v ₁ ,loc ₁ ,s],[v ₂ ,loc ₂ ,s],…,[v _n ,loc _n ,s]} _t (6) Where s∈S; v ₁ ,v ₂ ,...,v _n is a set of vehicles; loc ₁ ,loc ₂ ,...,loc _n is the set of vehicle locations requesting service s at time t; The action space is used to describe the actions taken when placing a service on an edge server; The action a taken at a given time t is as follows: Where π is the policy function required to generate actions for the observation set ω in time unit t; Indicates that service s is deployed on edge server e; Indicates that service s is not deployed on edge server e; The policy function π is a function executed by the actor network to map the state space to the action space, i.e., π:ω→a; The goal of the policy function π is to minimize the maximum edge resource usage and service delay, and to control the relative importance of resource usage and service delay by using the parameter β; The policy function π is expressed as follows: In the formula, β is the weight coefficient; The constraints of the policy function π include the mapping constraints Delay Constraint Resource Constraints 2. A dynamic service placement method based on edge computing and deep reinforcement learning in a connected vehicle network according to claim 1, characterized in that the network and service request related information includes edge server information, vehicle information, and service information; The edge server information includes an edge server set E, an edge server e, and a remaining resource capacity C _e of the edge server e; The vehicle information includes a vehicle set V; The service information includes the service set S, the number of vehicles λ _s requesting service s, the number of vehicles ε that can process one service instance at a time or provide parallel connections, the time t and vehicle location loc specified in the service request message, the amount of resources R _s consumed by the edge server deploying service s, and the delay requirement threshold D _s ; the service instances include media file downloading, cooperative awareness messaging, and environmental notification services in the Internet of Vehicles environment. 3. According to a method for dynamic service placement based on edge computing and deep reinforcement learning in a connected vehicle network according to claim 1, it is characterized in that the reward function is as follows: In the formula, is the immediate reward; γ is the reward coefficient; is the service delay at time t. 4. According to the method of dynamic service placement based on edge computing and deep reinforcement learning in the Internet of Vehicles in claim 1, it is characterized in that the loss function in the critic network training process is As shown below: Where θ is the critic network parameter; is the target value used to evaluate the quality of the strategy; _Qi (ω, a; θ) is the strategy quality of the service placement strategy; is the number of available resource units in the edge server. 5. According to the method of dynamic service placement based on edge computing and deep reinforcement learning in the Internet of Vehicles in claim 4, it is characterized in that the method of evaluating the policy quality of the service placement strategy by the critic network includes: judging the critic network loss function Whether it converges. If so, the evaluation passes. Otherwise, the evaluation fails.