CN112995951A - 5G Internet of vehicles V2V resource allocation method adopting depth certainty strategy gradient algorithm - Google Patents

Abstract

The present invention proposes a vehicle-to-vehicle (V2V) communication resource allocation method based on the Deep Deterministic Policy Gradient (DDPG) algorithm. The V2V communication uses the network slicing technology to access the 5G network, and uses the deep reinforcement learning optimization strategy to obtain the optimal V2V user. Channel allocation and transmit power joint optimization strategy, V2V users can reduce the mutual interference between V2V links by selecting appropriate transmit power and channels, and maximize the total system throughput of V2V links under the constraint of link delay. The invention can effectively solve the joint optimization problem of V2V user channel allocation and power selection by using the DDPG algorithm, and can perform stably in the optimization of a series of continuous action spaces.

Description

A 5G Internet of Vehicles V2V Resource Allocation Method Using Deep Deterministic Policy Gradient Algorithm

technical field

The present invention relates to an Internet of Vehicles technology, in particular to a resource allocation method for the Internet of Vehicles, and more particularly, to a vehicle-to-vehicle pairing system for 5G Internet of Vehicles using a Deep Deterministic Policy Gradient (DDPG) algorithm. Vehicle-to-Vehicle (V2V) communication resource allocation method.

Background technique

Vehicle-to-everything (V2X) is a typical application of Internet of Things (IoT) in the field of Intelligent Transportation System (ITS). The ubiquitous smart car network formed. The Internet of Vehicles shares and exchanges data according to agreed communication protocols and data exchange standards. It enables intelligent traffic management and services such as improved road safety, enhanced road condition awareness and reduced traffic congestion through real-time perception and collaboration between pedestrians, roadside facilities, vehicles, networks and the cloud.

Reasonable allocation of IoV resources is crucial for mitigating interference, improving network efficiency, and ultimately optimizing wireless communication performance. Most of the traditional resource allocation schemes use the slowly changing large-scale fading channel information for allocation. A heuristic location-dependent uplink resource allocation scheme has been proposed, which is characterized by spatial resource reuse without the need for complete channel state information, thus reducing signaling overhead. Another research has developed a framework including vehicle grouping, multiplexing channel selection and power control, which can reduce the total interference of V2V users to the cellular network while maximizing the sum rate or minimum achievable rate of V2V users. However, with the increasing communication volume and the substantial increase in communication rate requirements, high mobility leads to rapid changes in wireless channels, which brings great uncertainty to resource allocation. Traditional resource allocation methods cannot meet people's high reliability of the Internet of Vehicles. and low latency requirements.

Deep learning provides multi-layered computational models that can learn efficient data representations with multiple levels of abstraction from unstructured sources, providing a powerful data-driven approach to many traditionally considered difficult problems. The resource allocation scheme based on the deep reinforcement learning algorithm can better meet the requirements of high reliability and low latency of the Internet of Vehicles than the traditional resource allocation algorithm. Some literatures propose a novel distributed vehicle-to-vehicle communication resource allocation mechanism based on deep reinforcement learning that can be applied to unicast and broadcast scenarios. According to the distributed resource allocation mechanism, the agent, i.e. the V2V link or the vehicle does not need to wait for global state information to make a decision to find the optimal subband and transmit power level. However, the existing V2V resource allocation algorithms based on deep reinforcement learning cannot meet the needs of differentiated services in scenarios such as high bandwidth, large capacity, ultra-reliable and low-latency under 5G networks.

Therefore, the resource allocation method proposed in the present invention adopts the 5G network slicing technology, which can provide differentiated services for different application scenarios under the 5G network, and at the same time adopts the DDPG algorithm that can perform stable in the optimization of a series of continuous action spaces for V2V resource allocation, Taking the maximization of system throughput as the optimization goal of V2V resource allocation, it achieves a good balance between complexity and performance.

SUMMARY OF THE INVENTION

Purpose of the invention: In view of the above problems existing in the prior art, a method for V2V user resource allocation based on the deep reinforcement learning DDPG algorithm is proposed. V2V communication uses network slicing technology to access 5G network. The method can achieve V2V user resource allocation that maximizes system throughput with lower V2V link delay under the condition that the V2V link does not interfere with the V2I link.

Technical solution: In the case of considering the V2V link delay, the purpose of maximizing the throughput of the system communication system is achieved by reasonable resource allocation. We adopt 5G network slicing technology, V2V link and V2I link use different slices, V2V link does not interfere with V2I link. The distributed resource allocation method does not require the base station to centrally schedule channel state information, treats each V2V link as an agent, and selects the channel and transmit power based on the instantaneous state information and information shared from neighbors for each time slot. By establishing a deep reinforcement learning model, the DDPG algorithm is used to optimize the deep reinforcement learning model. According to the optimized deep reinforcement learning model, the optimal V2V user transmit power and channel allocation strategy are obtained. Completion of the above invention is achieved through the following technical solutions: a V2V resource allocation method based on 5G network slicing using the DDPG algorithm, including the following steps:

(1) The communication services in the Internet of Vehicles are divided into two types, namely broadband multimedia data transmission between vehicles and roadside facilities (V2I) and data transmission between vehicles and vehicles (V2V) about driving safety;

(2) Use 5G network slicing technology to divide V2I and V2V communication services into different slices respectively;

(3) The constructed user resource allocation system model is that K shares a channel with an authorized bandwidth of B for V2V users;

(4) Using a distributed resource allocation method, in the case of considering the V2V link delay, a deep reinforcement learning model is constructed with the goal of maximizing the throughput of the communication system;

(5) Consider the joint optimization problem in the continuous action space, and optimize the deep reinforcement learning model by using the deep deterministic policy gradient (DDPG) algorithm including three mechanisms of deep learning fitting, soft update, and memory playback;

(6) According to the optimized deep reinforcement learning model, the optimal V2V user transmit power and channel allocation strategy are obtained.

Further, the step (4) includes the following specific steps:

(4a), specifically define the state space S as the channel information related to resource allocation, including the instantaneous channel information G _t [m] of the corresponding V2V link of the sub-channel m, the interference intensity I _{t- 1} [m], the number of times the subchannel m is selected by the adjacent V2V link in the previous time slot N _t-1 [m], the remaining load L _t transmitted by the V2V user, and the remaining delay U _t , namely

s _t ={G _t , I _t-1 , N _t-1 , L _t , U _t }

Considering the V2V link as an agent, each V2V link selects the channel and transmit power based on the current state s _t ∈ S;

(4b), define the action space A as the transmit power and the selected channel, expressed as

为第k个V2V链路用户的发射功率，

为第m个信道被第k个V2V链路用户使用情况；in,

is the transmit power of the kth V2V link user,

is the usage of the mth channel by the kth V2V link user;

(4c), define the reward function R, the goal of V2V resource allocation is to select the spectral subband and transmit power of the V2V link, and maximize the V2V link under the requirement of satisfying the delay constraint and causing less interference to other V2V links. system throughput. So the reward function can be expressed as:

Among them, T ₀ is the maximum tolerable delay, λ _d and λ _p are the weights of the two parts, and T ₀ -U _t is the time used for transmission. As the transmission time increases, the penalty will also increase.

(4d), according to the established S, A and R, a deep reinforcement learning model is established on the basis of Q learning, and the evaluation function Q(s _t , a _t ) represents the discounted reward generated after the action a _t is executed from the state s _t , the Q value update function is:

Among them, r _t is the immediate reward function, γ is the discount factor, s _t is the state information of the V2V link at time t, s _t+1 represents the state of the V2V link after executing at _t , and A is composed of action at _t action space.

Beneficial effects: a V2V resource allocation method based on 5G network slicing using a deep deterministic strategy gradient algorithm proposed by the present invention, V2V communication uses network slicing technology to access 5G network, and uses deep reinforcement learning optimization strategy to obtain optimal V2V users Channel allocation and transmit power joint optimization strategy, V2V users can reduce the mutual interference between V2V links by selecting appropriate transmit power and allocation channels, and maximize the system throughput of V2V links under the constraint of link delay . The invention can effectively solve the joint optimization problem of V2V user channel allocation and power selection by using the DDPG algorithm, and can perform stably in the optimization of a series of continuous action spaces.

To sum up, in the case of ensuring reasonable resource allocation, low interference between V2V links and low computational complexity, a V2V resource allocation method based on 5G network slicing using a deep deterministic policy gradient algorithm proposed by the present invention is as follows: It is superior in terms of maximizing the throughput of the V2V system.

Description of drawings

1 is a flowchart of a 5G Internet of Vehicles V2V resource allocation method using a deep deterministic policy gradient algorithm provided by an embodiment of the present invention;

2 is a schematic diagram of a V2V user resource allocation model based on 5G network slicing technology provided by an embodiment of the present invention;

3 is a schematic diagram of a deep reinforcement learning framework based on an Actor-Critic model provided by an embodiment of the present invention;

4 is a schematic diagram of a deep reinforcement learning model for V2V communication provided by an embodiment of the present invention;

Detailed ways

The core idea of the present invention is: V2V communication uses network slicing technology to access 5G network, adopts distributed resource allocation method, treats each V2V link as an intelligent body, establishes a deep reinforcement learning model, and uses DDPG algorithm to optimize deep reinforcement Learning models. According to the optimized deep reinforcement learning model, the optimal V2V user transmit power and channel allocation strategy are obtained.

The present invention will be described in further detail below.

Step (1), the communication service in the Internet of Vehicles The communication service in the Internet of Vehicles is divided into two types, namely, broadband multimedia data transmission between vehicles and roadside facilities (V2I) and vehicle-to-vehicle (V2V) and driving. Security-Related Data Transmission.

In step (2), the 5G network slicing technology is used to divide V2I and V2V into different slices respectively.

In step (3), the constructed user resource allocation system model is that K shares a channel with an authorized bandwidth of B to V2V users,

It includes the following specific steps:

表示；(3a), establish a V2V user resource allocation system model, the system includes K pairs of V2V users (VUEs), represented by a set κ = {1, 2, ..., K}, the total authorized bandwidth B is equally divided into M bandwidths is the sub-channel of B ₀ , and the sub-channel uses a set

express;

(3b), the SINR of the K-th V2V link can be expressed as:

in,

是第k′条V2V链路对第k条V2V链路的干扰增益。第K条V2V链路的信道容量可以表示为：G _d is the total interference power of all V2V links sharing the same RB, g _k is the channel gain of the k-th V2V link IoV user,

is the interference gain of the k'th V2V link to the kth V2V link. The channel capacity of the Kth V2V link can be expressed as:

C ^v [k]=W·log(1+ ^γv [k]); Expression 3

(3c), for the kth V2V link, the channel information selected at time t is:

则第m个信道被第k条V2V链路使用，同时有

且i≠m，即

K为V2V链路总个数，M为V2V链路接入切片的可用信道总数。like

Then the mth channel is used by the kth V2V link, and there are

And i≠m, that is

K is the total number of V2V links, and M is the total number of available channels of the V2V link access slice.

In step (4), a distributed resource allocation method is adopted, and a deep reinforcement learning model is constructed with the goal of maximizing the throughput of the communication system under the consideration of the V2V link delay, including the following specific steps:

子信道前一时隙接收到的干扰强度I_t-1[m]，

信道m在前一时隙被相邻的V2V链路选择的次数N_y-1[m]，

剩余的V2V负载L_t，剩余时延U_t，即(4a), specifically define the state space S as the observation information related to resource allocation, including the instantaneous channel information of the corresponding V2V link of the sub-channel

The received interference strength It _-1 [m] of the subchannel in the previous slot,

Number N _y-1 [m] that channel m was selected by adjacent V2V links in the previous time slot,

The remaining V2V load L _t , the remaining time delay U _t , namely

(4b), define the action space A as the transmit power and the selected channel, expressed as

为第k个V2V链路用户的发射功率，

为第m个信道被第k个V2V链路用户使用情况，

表示第m个信道被第k个V2V链路用户使用，

表示第m个信道没有被第k个V2V链路用户使用；in,

is the transmit power of the kth V2V link user,

is the usage of the mth channel by the kth V2V link user,

indicates that the mth channel is used by the kth V2V link user,

Indicates that the mth channel is not used by the kth V2V link user;

(4c), define the reward function R, the goal of V2V resource allocation is to select the spectral subband and transmit power of the V2V link, and maximize the V2V link under the requirement of satisfying the delay constraint and causing less interference to other V2V links. system throughput. So the reward function can be expressed as:

Among them, T ₀ is the maximum tolerable delay, λ _d and λ _p are the weights of the two parts, and T ₀ -U _t is the time used for transmission. As the transmission time increases, the penalty will also increase. For good long-term returns, both immediate and future returns should be considered. Therefore, the main goal of reinforcement learning is to find a strategy that maximizes the expected cumulative discounted return,

where β∈[0,1] is the discount factor;

(4d), according to the established S, A and R, establish a deep reinforcement learning model on the basis of Q learning: the evaluation function Q(s _t , a _t ) represents the discounted reward generated after performing the action a _t from the state s _t , the Q value update function is

Among them, r _t is the immediate reward function, γ is the discount factor, s _t is the state information of the V2V link at time t, s _t+1 represents the state of the V2V link after executing at _t , and A is composed of action at _t action space.

Step (5), in order to solve the problem of V2V resource allocation based on 5G network slicing, the action space in the deep reinforcement learning model established with the V2V link as the agent includes two variables: transmission power and channel selection, considering a certain range of transmission power. In order to solve this high-dimensional action space, especially the joint optimization problem in the continuous action space, the deep reinforcement learning model is optimized by using the DDPG algorithm including the three mechanisms of deep learning fitting, soft update, and memory playback.

Deep learning fitting means that the DDPG algorithm is based on the Actor-Critic framework, and uses deep neural networks with parameters θ and δ to fit the deterministic strategy a=μ(s|θ) and the action value function Q(s, a|δ) As shown in Figure 3 of the accompanying drawings.

Soft update means that the parameters of the action value network are used to calculate the gradient of the policy network while the parameters of the action value network are updated frequently, which makes the learning process of the action value network likely to be unstable. Therefore, a soft update method is proposed to update the network.

Create two neural networks, the online network and the target network, for the policy network and the action-value network, respectively:

During the training process, gradient descent is used to continuously update the network. The update method of the target network is as follows

θ′=τθ+(1-τ)θ Expression 9

δ′=τδ+(1-τ)δ Expression 10

The experience replay mechanism means that the state transition sample data generated when interacting with the environment has a time-series correlation, which is easy to cause the deviation of the action value function fitting. Therefore, drawing on the experience playback mechanism of the DQN algorithm, the collected samples are first put into the sample pool, and then some mini-batch samples are randomly selected from the sample pool for training the network. This process removes the correlation and dependency between samples, solves the problem of correlation between data and its non-static distribution, and makes the algorithm easier to converge.

Using the DDPG algorithm including deep learning fitting, soft update, and memory playback to optimize the deep reinforcement learning model includes the following steps:

(5a), initialize the number of training rounds P;

(5b), initialize the time step t in the P round;

(5c), the online Actor policy network outputs the action at according to the input state s _t , and obtains the immediate reward r _t , and at the same time goes to the next state s _t+1 , thereby obtaining the training data (s _t , at _t , _{r t} , s _t+1 );

(5d), store the training data (s _t , at _t , r _t , s _t+1 ) into the experience playback pool;

(5e), randomly sample m training data (s _t , at , r _t , s _{t +1} ) from the experience playback pool to form a dataset, and send it to the online Actor policy network, online Critic evaluation network, and target Actor policy network and target Critic evaluation network;

(5f), set the Q estimate as

y _i =r _i +γQ′(s _i+1 , μ′(s _i+1 |θ′)|δ′) Expression 11

The loss function of the online critical evaluation network is defined as

Update all parameters θ of Critic's current network through gradient back-propagation of the neural network;

(5g), define the gradient of the sampling strategy for the online Actor strategy network as

Update all parameters δ of Actor's current network through gradient back-propagation of the neural network;

(5h), if the number of online training reaches the target network update frequency, update the target network parameters δ′ and θ′ respectively according to the online network parameters δ and θ;

(5i), judge whether t<K is satisfied, K is the total time step in p round, if so, t=t+1, go to step 5c, otherwise, go to step 5j;

(5j), judge whether p<I is satisfied, I is the set threshold for the number of training rounds, if so, p=p+1, go to step 5b, otherwise, the optimization ends, and the optimized deep reinforcement learning model is obtained.

Step (6), according to the optimized deep reinforcement learning model, obtain the optimal V2V user transmit power and channel allocation strategy, including the following steps:

(6a), using the deep reinforcement learning model trained by the DDPG algorithm, input the state information _sk (t) of the system at a certain time;

得到最优的V2V用户发射功率

和分配信道

(6b), output the optimal action strategy

Get the optimal V2V user transmit power

and assign channels

Finally, the drawings in the specification are described in detail.

In Figure 1, the flow of a 5G Internet of Vehicles V2V resource allocation method using a deep deterministic policy gradient algorithm is described. V2V communication uses network slicing technology to access the 5G network, and uses DDPG to optimize the deep reinforcement learning model to obtain the optimal V2V User channel allocation and transmit power joint optimization strategy.

In Figure 2, the V2V user resource allocation model based on 5G network slicing technology is described, and V2V communication and V2I communication use different slices.

In Figure 3, the deep learning fitting is described. The DDPG algorithm is based on the Actor-Critic framework and uses deep neural networks with parameters θ and δ to fit the deterministic policy a=μ(s|θ) and the action value function Q, respectively. (s, a|δ).

In Figure 4, a deep reinforcement learning model for V2V communication is depicted. It can be seen that the V2V link as an agent selects the channel and transmit power according to the reward function based on the current state s _t ∈ S.

According to the description of the present invention, those skilled in the art should not be difficult to see that the V2V resource allocation method based on the deep reinforcement learning DDPG algorithm using the 5G network slicing technology of the present invention can improve the system throughput and ensure that the communication delay meets the security requirements.

The contents not described in detail in the application of the present invention belong to the prior art known to those skilled in the art.

Claims (2)

1. A5G Internet of vehicles V2V resource allocation method adopting a depth deterministic strategy gradient algorithm is characterized by comprising the following steps:

(1) communication services in the internet of vehicles are divided into two types, namely broadband multimedia data transmission between vehicles and roadside facilities (V2I) and data transmission between vehicles (V2V) about driving safety;

(2) dividing V2I and V2V communication traffic into different slices respectively by using a 5G network slicing technology;

(3) the constructed user resource allocation system model is that K shares a channel with the authorized bandwidth of B for V2V users;

(4) by adopting a distributed resource allocation method, under the condition of considering V2V link delay, a deep reinforcement learning model is constructed with the aim of maximizing the throughput of a communication system;

(5) considering a joint optimization problem in a continuous action space, and optimizing a deep reinforcement learning model by using a deep certainty strategy gradient (DDPG) algorithm comprising three mechanisms of deep learning fitting, soft updating and memory playback;

(6) and obtaining the optimal V2V user transmitting power and channel allocation strategy according to the optimized deep reinforcement learning model.

2. The 5G Internet of vehicles V2V resource allocation method adopting the deep deterministic strategy gradient algorithm according to claim 1, wherein the step (4) comprises the following specific steps:

(4a) the state space S is specifically defined as observation information related to resource allocation, including subchannel m corresponding V2V link instantaneous channel state information G_t[m]Interference strength I received in the previous time slot of sub-channel m_t-1[m]Number of times N that subchannel m was selected by an adjacent V2V link in the previous time slot_t-1[m]Residual load L transmitted by V2V user_tResidual time delay U_tI.e. by

s_t＝{G_t，I_t-1，N_t-1，L_t，U_t}

Considering the V2V link as an agent, each time the V2V link is based on the current state s_tSelecting a channel and transmitting power by the E.S;

(4b) defining the action space A as the transmit power and the selected channel, denoted as

Wherein,

the transmission power of the k-th V2V link user,

for the use of the mth channel by the kth V2V link user,

indicating that the mth channel is used by the kth V2V link user,

indicating that the mth channel is not used by the kth V2V link user;

(4c) the goal of defining the reward function R, V2V resource allocation is that the V2V link selects the spectral sub-band and transmit power to maximize the system throughput of the V2V link while satisfying the delay constraint. The reward function can thus be expressed as:

wherein, T₀To the maximum tolerable delay, λ_d、λ_pIs a weight of two parts, T₀-U_tIs the time taken for transmission, the penalty will increase as the transmission time increases;

(4d) establishing a deep reinforcement learning model on the basis of Q learning according to the established S, A and R, and evaluating the function Q (S)_t，a_t) Represents the slave state s_tPerforming action a_tThe resulting discount reward, the Q-value update function is:

wherein r is_tIs an instant reward function, gamma is a discount factor, s_tFor the state of V2V link at time tInformation, s_t+1Indicating that the V2V link is performing a_tIn the latter state, A is action a_tThe formed motion space.

Images