CN111065102B - Q learning-based 5G multi-system coexistence resource allocation method under unlicensed spectrum - Google Patents

Abstract

The invention relates to a method for allocating 5G multi-system coexisting resources under an unlicensed spectrum based on Q learning, and belongs to the technical field of spectrum allocation in wireless communication. The method comprises the following steps: step 1: calculating the throughput of the NR-U system and the WiFi system in a coexistence scene, and respectively determining the requirements of the NR-U system and the WiFi system on frequency spectrum resources; step 2: calculating the optimal ABS number under the coexistence scene, specifically: determining an objective function of the optimal ABS number in a coexistence scene; and traversing the q values in sequence to find the q values which accord with the optimization target, and calculating the quantity of the ABS according to the calculated q values: n is a radical of_ABS(1-q) T; t is the length of a wireless frame in the 5G NR system; and step 3: and matching the ABS position in the coexistence scene, and matching the WiFi system with the ABS position by using Q learning. The method ensures the fairness of the system and simultaneously effectively improves the total throughput of the heterogeneous network system; the utilization rate of the frequency spectrum resources of the coexisting system is effectively improved.

Description

Q learning-based 5G multi-system coexistence resource allocation method under unlicensed spectrum

Technical Field

The invention provides a Q-learning-based method for allocating 5G multi-system coexistence resources under an unlicensed spectrum, and belongs to the technical field of spectrum allocation in wireless communication.

Background

As mobile communication is about to advance into the fifth generation, emerging services based on 5G networks, such as interactive games, virtual/augmented reality technologies, and telemedicine services, are emerging in a blowout manner. Meanwhile, with the rapid development and application popularization of new technologies such as global industrial internet or internet of things, the requirement for industrial technology upgrading is far higher than that of the traditional internet or internet of things application. In order to deal with the explosively increasing mass data, how to scientifically plan and efficiently utilize spectrum resources to improve the network capacity is an important way for solving the problem.

The 5G network is a heterogeneous system with multi-network integration and coexistence, the network types are various and large-scale connection is supported, and the unlicensed spectrum is urgently needed to be used for sharing mobile communication services. However, the coexistence problem in the unlicensed frequency band is further complicated due to the non-uniformity of the wireless access standards caused by the heterogeneity of the transmission technologies of the lower layers of different networks, the difference of the transmission power, and the affiliation to different operators. When the 5G system operates in the Unlicensed spectrum (5th Generation Mobile Networks New Radio in Unlicensed,5G NR-U), the interference strength in the dense networking environment of the Unlicensed frequency band is increased, and especially, the direct interference is caused to the existing WiFi system on the Unlicensed frequency band. Therefore, how to solve the coexistence of the 5G NR-U system and the WiFi system on the unlicensed frequency band is an urgent problem to be solved.

Currently, a coexistence scheme based on Almost Blank Subframes (ABS) is mainly used for the above problem. The coexistence scheme based on the ABS is inspired by using the ABS for enhanced inter-cell interference coordination proposed in LTE Release10, and is improved on the basis of the ABS, so that the coexistence of an NR-U system and a WiFi system is realized. In addition, the existing ABS coexistence schemes statically allocate blank subframes for WiFi, and the traffic load of WiFi and other factors are not considered, thereby causing the waste of ABS subframes. Therefore, how to allocate an appropriate amount of ABS to WiFi according to the traffic load of WiFi is an urgent problem to be solved.

In addition, considering that the NR-U system and the WiFi system are heterogeneous systems, no information interaction is possible between the two systems. However, machine learning can make a prediction or an optimal decision about the current environment or state, and does not require data interaction between communication devices, and is therefore well suited for the communication environment of heterogeneous networks. Therefore, the problem of coexistence of the NR-U system and the WiFi system on the unlicensed frequency band can be effectively solved by applying the machine learning method.

Disclosure of Invention

The invention aims to solve the technical defect that an NR-U system and a WiFi system on an unlicensed frequency band cannot coexist on the premise of no constraint, and provides a 5G multi-system coexistence resource allocation method under an unlicensed frequency spectrum based on Q learning.

The purpose of the invention is mainly realized by the following technical scheme:

the 5G multi-system coexistence resource allocation method comprises three parts, namely throughput calculation, ABS number calculation and ABS position matching;

the method comprises the following steps of:

step 1: calculating the throughput of the NR-U system and the WiFi system in a coexistence scene;

wherein, the throughput of the WiFi system is (1):

wherein, P_trAnd P_sIndicating the probability of successful transmission of a user in the channel and the probability of at least one user in the channel being transmitted, E P]Representing the average data transmitted in a time slot, T_σ，T_sAnd T_cRespectively representing the average time of idle time slots, the average time of successful data transmission and the average time of collision occurrence; p_tr、P_s、T_sAnd T_cExpressed as (2), (3), (4) and (5), respectively:

P_tr＝1-(1-τ)^N (2)

T_s＝H+E[P]+SIFS+δ+ACK+DIFS+δ (4)

T_c＝H+E[P]+DIFS+δ (5)

wherein N is the number of WiFi STAs in a coexistence scenario, H is the length of the MAC and PHY layer headers, δ is transmission delay, ACK, DIFS, and SIFS respectively represent the inter-frame spacing, the acknowledgement frame time, and the short inter-frame spacing of the DCF, and τ is the transmission probability of each WiFi STA in any time slot, and is represented as (6):

wherein CW_minDenotes the size of the minimum contention window, m denotes the maximum backoff state, p_cRepresenting the probability of collision of data frames in the channel, by (7):

p_c＝1-(1-τ)^N-1 (7)

the throughput of the NR-U system is (8):

wherein B is the sub-bandwidth of one channel in the unlicensed frequency band, gamma_mThe SINR for the mth UE can be expressed as γ_m＝p_m|g_m|²/(I+r)，p_mFor the transmission power of the mth UE, g_mThe channel gain of the mth UE is obtained, I is interference power, and r is the power of white noise;

wherein, the User Equipment is called User Equipment Devices, abbreviated as UE;

step 2: calculating the optimal ABS number in the coexistence scene, specifically comprising the following substeps:

step 2.1, traversing all values of q in the objective function (9) in sequence to find a q value which accords with the optimization target:

wherein the constraint condition q belongs to [0.1,0.2,. ], 0.9 ∈]Is to ensureThe number of ABS allocated for WiFi is proved to be integral multiple of the number of subframes; s_totalThe total throughput of the system, namely the sum of the throughputs of an NR-U system and a WiFi system, q is the time proportion occupied by the NR-U system, (1-q) is the silence time of the NR-U system, namely the duration of ABS, lambda is a weight factor for measuring the throughputs and fairness, and F_qDenotes the fairness index of the NR-U system with the WiFi system, denoted (10):

step 2.2 calculating the quantity N of ABS according to the q value found in step 2.1_ABSSpecifically, the method is realized by formula (11):

N_ABS＝(1-q)T (11)

wherein, T is the length of a wireless frame in the 5G NR system;

and step 3: the method comprises the following steps of matching ABS positions in a coexistence scene, specifically using Q learning to realize the matching of a WiFi system and the ABS positions, and comprising the following steps:

step 3.1) the NR-U system as an intelligent agent in Q learning obtains the minimum value s of WiFi throughput in ABS duration by monitoring the network environment parameters of the WiFi system_minAnd a maximum value s_max；

Step 3.2) initializing a Q value table, and randomly selecting a state s from the state space s_tAnd from action set a ═ a_t},t∈[0,10-N_ABS]Randomly selecting an action a_t；

Wherein the state space s is represented as (12):

wherein s is_wAverage throughput for WiFi over ABS duration;

step 3.3) performing action a_tObtain its reported value r and observe the next state a_t+1；

Wherein the return value r is defined as the state of the agents_tTake action a_tThe prize value that can be achieved, is denoted (13):

wherein the content of the first and second substances,

representing the throughput of the WiFi system during the duration corresponding to the ith ABS, σ is set to avoid a denominator of 0; when in use

And s_maxThe closer the return value is, the higher the return value is; on the contrary, when

And s_maxThe larger the gap, the lower the return value;

step 3.4) updating the Q value table according to the step (14), and updating the current state s_t＝s_t+1；

Q(s_t,a_t)＝(1-α)Q(s_t,a_t)+α[r+γmax_aQ(s_t+1，a)]

(14)

Wherein, Q(s)_t,a_t) Is shown in the current state s_tNext, the agent performs action a_tObtaining accumulated decision information;

step 3.5) finding the current state s_tThe lower corresponds to the best Q value and the corresponding action is the position of the ABS.

Therefore, coexistence of the Q-learning-based NR-U system and the WiFi system is completed, and the Q-learning-based 5G multi-system coexistence resource allocation method under the unlicensed spectrum is realized.

Advantageous effects

Compared with the prior art, the method for allocating the 5G multi-system coexisting resources under the unlicensed spectrum based on Q learning has the following characteristics:

(1) the coexistence resource allocation method can determine the appropriate ABS number according to the load of WiFi while maximizing the system throughput, and compared with the existing static ABS allocation scheme, the method can effectively improve the total throughput of the heterogeneous network system while ensuring the system fairness;

(2) according to the method, the transmission position of the WiFi is matched with the ABS position by using the Q learning algorithm, so that the frequency spectrum resource utilization rate of the coexistence system can be effectively improved.

Drawings

Fig. 1 is a co-existence system model of a 5G multi-system co-existence resource allocation method under a Q-learning-based unlicensed spectrum and an NR-U system and a WiFi system in embodiment 1;

fig. 2 is a schematic diagram of a 5G multi-system coexistence intelligent resource allocation method based on Q learning under an unlicensed spectrum and a schematic diagram of a WiFi system data transmission position and an ABS position in embodiment 1 under a mismatch condition.

Detailed Description

The following describes the content of the method for allocating 5G multi-system coexistence resources under the Q-learning-based unlicensed spectrum in detail with reference to the accompanying drawings.

Example 1

When the method for allocating the 5G multi-system coexistence resources under the unlicensed spectrum based on Q learning is implemented specifically, the scenario shown in fig. 1 is based on. As shown in fig. 1, there are 2 UEs in the NR-U system and 2 STAs in the WiFi system, and in the specific implementation, 10 UEs and 8 STAs share the same unlicensed spectrum resource without loss of generality.

But the NR-U system has better interference tolerance because the MAC mechanism of centralized scheduling is adopted. And the WiFi system adopts a MAC mechanism based on competition, and when the aim of maximizing the throughput of the coexistence system is taken as an optimization target, the resources of the unlicensed frequency band are distributed to the NR-U system as much as possible, so that the WiFi cannot access a channel for communication, and the throughput of the WiFi system is greatly reduced. In addition, the NR-U system in the coexistence system model occupies the whole unlicensed frequency band, and the WiFi system can only use the unlicensed frequency band for communication in the silent period of the NR-U system.

Therefore, the method of the invention allocates reasonable ABS number for the WiFi system by calculating the throughput of the NR-U system and the WiFi system on the premise of ensuring the maximization of the system throughput and the fairness of the NR-U system and the WiFi system.

The intelligent resource allocation method for 5G multi-system coexistence under the Q learning-free spectrum comprises the following steps:

step A: calculating the throughput of the NR-U system and the WiFi system in a coexistence scene;

in this embodiment, the number of UEs in the NR-U system is 10, the number of STAs in WiFi is 8, and p can be obtained by combining (6) and (7)_cAnd the value of τ. Then, the value of τ is substituted into (2) and (3) to obtain P_trAnd P_sThe value of (c). In the embodiment of the present invention, H is PHY header + MAC header, where PHY header is 192bits, and MAC header is 224 bits. Furthermore, E [ P ]]8224bits, SIFS 16 μ s, DIFS 34 μ s, δ 9 μ s, ACK 112bits + PHY header. Substituting the values into (4) and (5) can obtain T_sAnd T_cThe value of (c). Finally, P is added_tr、P_s、E[P]、T_σ＝20μs、T_sAnd T_cThe throughput of WiFi can be obtained by substituting in (1).

For the throughput of NR-U systems, p is_mSubstitution of 20dBm, I-95 dBm and a path loss model of 15.3+50log (100) into γ_m＝p_m|g_m|²V (I + r) is determined by_mThen, the throughput of the NR-U system can be obtained by substituting the value of (8).

And B: calculating the optimal ABS number in a coexistence scene;

in this embodiment, the specific algorithm flow for traversing and solving the optimal ABS number in the coexistence scene is as follows: (1) setting the value range of q as [0.1,0.9 ]]The search step length is 0.1; (2) the initialization q is 0.1,

q^*when it is 0, calculate U_q＝λ(qS_L+(1-q)S_W)+(1-λ)F_q(ii) a (3) If it is not

Then

q^*Q; (4) jumping to the next q, q being q + 1; (5) until q is 0.9, return q^*And

the optimal ABS number can be obtained through the traversal algorithm.

And C: ABS position matching under a coexistence scene;

fig. 2 shows a schematic diagram of the data transmission position and the ABS position of the WiFi system in case of mismatch. It can be seen that the WiFi system has data to transmit when the WiFi system is not in blank subframes, and since the spectrum resource is being used by the NR-U system at this time, the WiFi system detects that the channel is in a busy state and starts a backoff mechanism. At this time, if the backoff time is within the ABS, the WiFi system cannot occupy the channel for communication due to the backoff state, which not only wastes spectrum resources but also reduces the throughput of the system. Because, when considering the coexistence of the NR-U system and the WiFi system using the ABS mechanism, in addition to configuring the appropriate number of ABSs, the ABS needs to be placed at an appropriate position according to the data transmission situation of the WiFi system.

The Q-learning-based intelligent resource allocation method for 5G multi-system coexistence under the unlicensed spectrum can match the proper ABS position according to the data transmission rule of WiFi, so that coexistence of the WiFi and the NR-U system is realized.

Therefore, the method for allocating the 5G multi-system coexistence resources under the unlicensed spectrum based on Q learning is completed.

Those skilled in the art will appreciate that the parameters are not limited to the specific assignment in the embodiment, and those skilled in the art can make the specific assignment according to the specific application scenario and the physical meaning of the parameters.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (3)

1. A method for allocating 5G multi-system coexisting resources under an authorization-free spectrum based on Q learning is characterized in that: the method comprises the following steps:

step 1: calculating the throughput of the NR-U system and the WiFi system in a coexistence scene;

step 2: calculating the optimal ABS number under the coexistence scene, and the steps are as follows:

step 2.1, traversing all values of q in the objective function (9) in sequence to find a q value which accords with the optimization target:

wherein the constraint condition q belongs to [0.1,0.2,. ], 0.9 ∈]The number of ABS allocated for WiFi is integral multiple of the number of sub-frames; s_totalIs the total throughput of the system, S, which is the sum of the throughput of the NR-U system and the WiFi system_LIs the throughput, s, of the NR-U system_wAverage throughput for WiFi over ABS duration; q is the time proportion occupied by the NR-U system, (1-q) is the silence time of the NR-U system, namely the duration of ABS, N is the number of STAs in WiFi in a coexistence scene, M is the total number of User Equipment (UE), lambda is a weight factor for measuring throughput and fairness, and F_qDenotes the fairness index of the NR-U system with the WiFi system, denoted (10):

step 2.2 calculating the number N of ABS from the q value calculated in step 2.1_ABSSpecifically, it is calculated by equation (11):

N_ABS＝(1-q)T (11)

wherein, T is the length of a wireless frame in the 5G NR system;

and step 3: the method comprises the following steps of matching ABS positions in a coexistence scene, specifically using Q learning to realize the matching of a WiFi system and the ABS positions, and comprising the following steps:

step 3.1) the NR-U system as an intelligent agent in Q learning obtains the minimum value s of WiFi throughput in ABS duration by monitoring the network environment parameters of the WiFi system_minAnd a maximum value s_max；

Step 3.2) initializing a Q value table, and randomly selecting a state s from the state space s_tAnd from action set a ═ a_t},t∈[0,10-N_ABS]Randomly selecting an action a_t；

Wherein the state space s is represented as (12):

wherein s is_wAverage throughput for WiFi over ABS duration;

step 3.3) performing action a_tObtain its reported value r and observe the next state a_t+1；

Wherein the return value r is defined as the state s of the agent_tTake action a_tThe prize value that can be achieved, is denoted (13):

wherein the content of the first and second substances,

representing the throughput of the WiFi system during the duration corresponding to the ith ABS, σ is set to avoid a denominator of 0; when in use

And s_maxThe closer the return value is, the higher the return value is; on the contrary, when

And s_maxThe larger the gap, the lower the return value;

step 3.4) updating the Q value table according to the step (14), and updating the current state s_t＝s_t+1；

Q(s_t,a_t)＝(1-α)Q(s_t,a_t)+α[r+γmax_aQ(s_t+1，a)] (14)

Wherein, Q(s)_t,a_t) Is shown in the current state s_tNext, the agent performs action a_tObtaining accumulated decision information; alpha is the learning rate and gamma is the discount factor;

step 3.5) finding the current state s_tThe lower corresponds to the best Q value and the corresponding action is the position of the ABS.

2. The method according to claim 1, wherein the method for allocating the 5G multi-system coexistence resources under the Q-learning-free unlicensed spectrum comprises: the throughput of the WiFi system in step 1 is (1):

wherein, P_trAnd P_sIndicating the probability of successful transmission of a user in the channel and the probability of at least one user in the channel being transmitted, E P]Representing the average data transmitted in a time slot, T_σ，T_sAnd T_cRespectively representing the average time of idle time slots, the average time of successful data transmission and the average time of collision occurrence; p_tr、P_s、T_sAnd T_cExpressed as (2), (3), (4) and (5), respectively:

P_tr＝1-(1-τ)^N (2)

T_s＝H+E[P]+SIFS+δ+ACK+DIFS+δ (4)

T_c＝H+E[P]+DIFS+δ (5)

wherein N is the number of WiFi STAs in a coexistence scenario, H is the length of the MAC and PHY layer headers, δ is transmission delay, ACK, DIFS, and SIFS respectively represent the inter-frame spacing, the acknowledgement frame time, and the short inter-frame spacing of the DCF, and τ is the transmission probability of each WiFi STA in any time slot, and is represented as (6):

wherein CW_minDenotes the size of the minimum contention window, m denotes the maximum backoff state, p_cRepresents the probability of collision of data frames in the channel, and can be expressed as (7):

p_c＝1-(1-τ)^N-1 (7) 。

3. the method according to claim 1, wherein the method for allocating the 5G multi-system coexistence resources under the Q-learning-free unlicensed spectrum comprises: the throughput of the NR-U system in step 1 is (8):

wherein B is the sub-bandwidth of one channel in the unlicensed frequency band, gamma_mThe SINR for the mth UE can be expressed as γ_m＝p_m|g_m|²/(I+r)，p_mFor the transmission power of the mth UE, g_mThe channel gain of the mth UE is obtained, I is interference power, and r is the power of white noise;

the UE is called User Equipment Devices, and is abbreviated as UE.

Images