CN112867117B - A Q-learning-based energy-saving method in NB-IoT - Google Patents

Abstract

The invention relates to a Q-learning-based energy-saving method in NB-IoT, and belongs to the technical field of communication. In this method, the base station can dynamically control the number of devices that initiate random access in each transmission time interval according to parameters such as network load, repetition times, transmission data resources, etc., so as to reduce the number of devices that collide during the random access process. Reduce the total energy consumption of random access to achieve the purpose of energy saving. In this method, the base station acts as an agent, the agent action is defined as the ratio of the number of devices allowed to initiate random access to the total number of active devices, and the agent state mainly includes a set of observed information, such as the number of successful communication devices and energy consumption, etc. The invention can reduce the energy consumption of the system and prolong the life of the equipment while ensuring the throughput of the equipment.

Description

An energy-saving method based on Q-learning in NB-IoT

technical field

The invention belongs to the field of communication technologies, and relates to a Q-learning-based energy-saving method in NB-IoT.

Background technique

In the past few decades, with the onset of the Industrial Revolution, humanity has made tremendous progress. In recent years, with the 5th-Generation (5G), the rapid development of the Internet of Things (IoT) and mobile computing, the Low Power Internet of Wide Area Network (LPWAN) has become increasingly popular. focus on. Narrow Band IoT (NB-IoT) based on cellular communication technology is a promising LPWAN technology. NB-IoT requires a minimum system bandwidth of 180kHz for downlink and uplink respectively. It is a new 3rd Generation Partnership Project (3GPP) radio access technology with low power consumption, narrow bandwidth, With the characteristics of strong coverage, low cost and massive connections, NB-IoT can be widely used in scenarios with poor channel transmission conditions (such as underground parking lots) or devices with delay tolerance (such as water meters).

The data communication of NB-IoT mainly occurs in the uplink channel, and the uplink channel resources mainly include Narrowband Physical Random Access Channel (NPRACH) and Narrowband Physical Uplink Sharing Channel (NPUSCH). Among them, the NPRACH is mainly used to start the random access process, and the NPUSCH is mainly responsible for the data transmission between the device and the base station. NB-IoT is mainly aimed at periodic devices with high latency tolerance (such as water meters, electricity meters, etc.). How to complete device communication with lower energy consumption while ensuring the throughput of the device is a problem we consider.

At present, the existing energy-saving mechanisms lack the process of dynamic learning. These mechanisms mainly optimize the back-off time of each device and the scalable discontinuous reception mechanism to achieve a compromise between device energy consumption and delay. However, in actual situations, there are scenarios where multiple devices transmit data at the same time, rationally schedule devices, control the number of device accesses in each Transmission Time Interval (TTI), and optimize communication on the premise of ensuring throughput. total energy consumption. Therefore, a scheduling mechanism based on the Q-learning algorithm is designed, so that the base station allows an appropriate amount of devices to perform random access in each TTI according to parameters such as network load, uplink resource quantity, data size, and repetition times, on the premise of ensuring throughput. It reduces system energy consumption and increases equipment life.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a Q-learning-based energy saving method in NB-IoT. The base station uses the Q-learning algorithm to flexibly adjust the number of devices initiating access requests in each TTI according to factors such as network load, delay, number of uplink resources, preamble size, and repetition times. consumption and prolong the life of the equipment. The method has the characteristics of simplicity and efficiency, and at the same time, it has certain portability.

In order to achieve the above object, the present invention provides the following technical solutions:

A Q-learning-based energy-saving method in NB-IoT, which includes the following steps:

S1: Define the state set and action set of the base station;

S2: At time t=0, the state and behavior Q value of the initialized base station is "0";

S3: Calculate the state value of the initial state _st of the base station;

S4: Choose an action at (i) according to the _ε -greedy strategy;

S5: After executing the behavior at (i), the system will obtain the environmental reward value _rt according to the reward function formula, and then enter the next state s _{t +1} ;

S6: Update the behavioral Q-value function of the base station according to the formula;

S7: t←t+1, jump to step S2.

Further, in step S1, the state set of the base station is represented as a series of previously observed information, namely S ^t ={U ^t-1 , U ^t-2 , U ^t-3 , LU ¹ }, where

表示随机接入能耗，

表示设备等待能耗，

表示数据传输能耗，

表示等待设备数量，

表示通信设备数量，

表示接入失败设备数量。in,

represents the energy consumption of random access,

Indicates that the device is waiting for power consumption,

represents the energy consumption of data transmission,

Indicates the number of waiting devices,

represents the number of communication devices,

Indicates the number of devices that fail to access.

For the behavior set, the ratio of the number of devices allowed to initiate random access in each TTI to the total active devices in the current TTI is taken as the base station behavior, and the base station behavior α in any t-th TTI is defined according to the Markov process of the limited action set ^t ∈ {0.2, 0.4, 0.6, 0.8, 1.0}.

Further, in step S2, the state of the base station and the Q value of the matrix are set to zero. The goal of solving the Markov decision process of the base station is to find an optimal strategy π ^* so that the value V(s) of each state s reaches the maximum at the same time. The state value function is represented as follows:

Among them, r(s _t , at _t ) represents the reward value that the base station obtains from the environment, and p(s _t+1 |s _t , at _t ) means that when the base station is in state s _t , it selects behavior at _t and then transfers to state s _{t +1} probability.

Further, in step S4, the base station's goal is to obtain a higher reward value, so in each state, an action with a higher Q value will be selected. However, in the initial stage of learning, the experience of state-action is relatively small, and the Q value cannot accurately represent the correct optimal value. The action of the highest Q value causes the base station to always follow the same path and it is impossible to explore other better values, so it is easy to fall into a local optimum. Therefore, the ε greedy strategy is introduced, and its main principles are as follows:

The agent randomly selects actions with probability ε, and chooses the action that maximizes the Q value with probability 1-ε.

Further, in step S5, after the base station performs the selected behavior, it will obtain a reward value from the environment, and the reward value function is defined as:

表示服务设备数，N表示总传输设备个数，T表示TTI个数，E^t表示第t个TTI中的系统总能耗。

represents the number of service devices, N represents the total number of transmission devices, T represents the number of TTIs, and E ^t represents the total system energy consumption in the t-th TTI.

in,

n ^t represents the number of access devices allowed by the current TTI, r represents the number of repetitions, μ represents transmission data resources, Q represents total uplink resources, and ^mi represents the number of preambles.

E ^t = E _{sy, t} + E _{ra, t} + E _{wait, t} + E _{dt, t}

E _{sy, t} represents the synchronization energy consumption, E _{ra, t} represents the random access energy consumption, E _{wait, t} represents the device waiting energy consumption, and E _{dt, t} represents the data transmission energy consumption.

Further, in step S6, after the base station obtains the reward value from the environment, it needs to update the Q matrix, and the update formula is:

where α represents the learning rate and 0<α<1, and Y represents the discount factor and 0≤Y<1. The learning rate and the discount factor work together to adjust the update of the Q matrix, which in turn affects the learning performance of the Q algorithm. α is 0.01, and Y is 0.8.

The beneficial effect of the present invention is that: through the Q learning algorithm, the system energy consumption can be reduced under the condition of ensuring the equipment throughput.

Description of drawings

Figure 1 is a model diagram of the Q-learning and environment interaction process;

Figure 2 is a step diagram of an energy-saving algorithm based on Q-learning;

Figure 3 is a flowchart of NB-IoT uplink communication.

Detailed ways

The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Aiming at the energy consumption problem of the NB-IoT system, the present invention proposes a Q-learning-based energy-saving method in the NB-IoT. Compared with the traditional optimization algorithm, the Q-learning algorithm in the present invention can dynamically optimize the transmission device, and the base station can flexibly adjust the number of access devices according to the real-time network situation. The process is shown in Figure 1. First, in a certain state, the base station performs a certain behavior based on the ε greedy strategy according to the current environment; then observe the environment to obtain the reward value, update the Q function value according to the formula, and determine the behavior of the next state. , and repeat the above actions until convergence.

The specific algorithm steps are shown in Figure 2. In the iterative process of the Q-learning algorithm, the state set is defined as S. If the decision time is t, then s _t ∈ S, indicating that the state of the base station at time t is s _t . At the same time, we define the limited set of actions that the base station may perform as A, where at ∈ A represents the base station’s behavior at time _t . The reward function r(s _t , at _t ) represents the reward value obtained from the environment after the base station performs the behavior a _t based on the state s _t it is in, and then transfers from the state s _t to s _t+1 , at the next decision time t+ 1 to update the _Qt function. Repeat the above steps until the iteration ends.

The flow of NB-IoT in the uplink transmission process is shown in Figure 3. First, the base station sends a narrowband primary synchronization signal (NPSS) and a narrowband secondary synchronization signal (NSSS) to the device to synchronize the time and frequency of the device and the cell. This process is a synchronization process. Then the base station The access request information is sent through NPRACH. After receiving the request information from the device, the base station responds through the Narrowband Physical Downlink Control channel (NPDCCH), and then the base station establishes a connection with the device. After the connection is established, the base station sends the scheduling request and data transmission via NPUSCH.

The Q-learning algorithm is actually a variation of Markov Decision Processes (MDP). In the energy-saving algorithm in NB-IoT, based on the working principle of the Q-learning algorithm, we express the state set as follows:

S ^t = {U ^t-1 , U ^t-2 , U ^t-3 , LU ¹ }, where

表示随机接入能耗，

表示设备等待能耗，

表示数据传输能耗，

表示等待设备数量，

表示通信设备数量，

表示接入失败设备数量。in,

represents the energy consumption of random access,

Indicates that the device is waiting for power consumption,

represents the energy consumption of data transmission,

Indicates the number of waiting devices,

represents the number of communication devices,

Indicates the number of devices that fail to access.

Taking the ratio of the number of devices allowed to initiate random access in each TTI to the total active devices in the current TTI as the base station behavior, the base station behavior set A={a(1), a(2), L, a(k) }. Any t TTI base station behaviors α ^t ∈ {0.2, 0.4, 0.6, 0.8, 1.0} are defined according to a Markov process with a finite set of actions.

The task faced by the base station is to decide an optimal strategy that maximizes the reward obtained. The base station will make the best decision on the next state/action based on the current state and environment. The discounted cumulative reward value function of state st can be expressed as:

Among them, r(s _t , at _t ) represents the instant reward obtained by the base station when the base station selects the action at in the state _{s t} . Υ represents a discount factor and 0≤Υ<1, and the discount factor tends to 0, indicating that the base station mainly considers immediate rewards. p(s _t+1 |s _t , at _t ) represents the probability of transitioning from state s _t to s _{t +1} when the base station selects action at . The goal of the MDP solution is to find an optimal policy π ^* that makes the value V(s) of each state s reach the maximum at the same time. According to Bellman's principle, when the total discounted expected reward of the base station is the largest, we can at least get an optimal policy π ^* such that:

where V ^* (s _t ) represents the maximum discounted cumulative reward value obtained by the base station starting from state s _t and following the optimal policy π ^* . For a given policy π, is the function that maps the state space to the action space, ie: π: s _t → a _t . Therefore, the optimal strategy can be expressed as:

π ^* (s _t )=argV ^* (s _t )

The goal of the base station is to obtain a high reward value, so in each state, an action with a high Q value will be selected. However, in the initial stage of learning, the experience of state-action is relatively small, and the Q value cannot accurately represent the correct optimal value. The action of the highest Q value causes the base station to always follow the same path and it is impossible to explore other better values, so it is easy to fall into a local optimum. Therefore, in order to overcome this shortcoming, the base station must select actions randomly, therefore, an ε-greedy strategy is introduced to reduce the possibility that the action selection strategy of the base station falls into a local optimal solution.

The agent randomly selects actions with probability ε, and chooses the action that maximizes the Q value with probability 1-ε.

Further, in step S5, after the base station performs the selected behavior, it will obtain a reward value from the environment, and the reward value function is defined as:

表示服务设备数，N表示总传输设备个数，T表示TTI个数，E^t表示第t个TTI中的系统总能耗。

represents the number of service devices, N represents the total number of transmission devices, T represents the number of TTIs, and E ^t represents the total system energy consumption in the t-th TTI.

in,

n ^t represents the number of access devices allowed by the current TTI, r represents the number of repetitions, μ represents transmission data resources, Q represents total uplink resources, and ^mi represents the number of preambles.

E ^t = E _{sy, t} + E _{ra, t} + E _{wait, t} + E _{dt, t} .

E _{sy, t} represents the synchronization energy consumption, E _{ra, t} represents the random access energy consumption, E _{wait, t} represents the device waiting energy consumption, and E _{dt, t} represents the data transmission energy consumption.

In the Q-learning algorithm, based on the policy π, the base station calculates the Q-value function recursively at each TTI as follows:

Obviously, the Q value represents the expected discounted reward obtained by the base station following policy π to perform action a _t when it is in state s _t . Therefore, our goal is to evaluate the Q value under the optimal policy π ^* . From the above formula, it can be concluded that the relationship between the state value function and the behavior value function is as follows:

However, based on the non-deterministic environment, the above Q-value function is only valid under the optimal strategy, that is, the value of the Q-value function changes (or does not converge) through Q-learning under the non-optimal strategy. Therefore, the calculation formula of the modified Q-value function is as follows:

Where α represents the learning rate and 0<α<1, the larger the learning rate, the less the effect of retaining the previous training. If each state-action pair can be repeated multiple times, the learning rate drops according to an appropriate scheme, and the Q-learning algorithm can converge to the optimal policy for any finite MDP. Υ represents the discount rate and 0<Υ<1, where Υ represents the importance of future rewards. Higher Υ values can capture long-term effective rewards, while lower Υ values make the agent focus more on immediate rewards. The learning rate and the discount factor work together to adjust the update of the Q matrix, which in turn affects the learning performance of the Q-learning algorithm.

Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should Various changes may be made in details without departing from the scope of the invention as defined by the claims.

Claims (1)

1. A method for Q-learning based energy saving in NB-IoT, the method comprising the steps of:

   s1: defining a set of states and a set of actions for a base station, a set of states being defined as a series of previously observed information, i.e. S^t＝{U^t-1,U^t-2,U^t-3,L U¹Therein of

   

   

   Which represents the energy consumption for random access,

   

   indicating that the device is waiting for energy consumption,

   

   which represents the energy consumption for data transmission,

   

   indicating the number of devices waiting for the device,

   

   which is indicative of the number of communication devices,

   

   representing the number of access failure devices, and defining an action set as the proportion of the number of devices which are allowed to initiate random access in each TTI to the total active devices in the current TTI;

   s2: setting the state and behavior Q value of the base station as a zero matrix at the moment when t is 0;

   s3: selecting an action a according to an epsilon greedy method_t(i) The method comprises the following steps In the initial stage of learning, the experience on state-action is less, the Q value cannot accurately represent the correct optimal value, the action with the highest Q value causes the base station to always follow the same path and cannot search other better values, so that the base station is easy to fall into local optimization, an epsilon greedy strategy is introduced, an intelligent body randomly selects action according to the probability of epsilon, and selects the action which enables the Q value to be maximum according to the probability of 1-epsilon, namely the action

   

   S4: performing an action a_t(i) Then, the system obtains the environment reward value R according to the formula_tThen enters the next state s_t+1: the reward value function is defined as:

   

   wherein

   

   Representing the number of serving devices, N representing the total number of transmission devices, T representing the number of TTIs, E^tIndicating the total energy consumption of the system in the t-th TTI,

   

   n^trepresenting the number of access devices allowed in the current TTI, r representing the number of repetitions, m representing the transmission data resource, Q representing the total uplink resource, mⁱIndicates the number of preambles, E^t＝E_sy,t+E_ra,t+E_wait,t+E_dt,t，E_sy,tIndicating synchronous energy consumption, E_ra,tIndicating random access power consumption, E_wait,tIndicating waiting energy consumption of the apparatus, E_dt,tRepresenting data transmission energy consumption;

   s5: and updating a behavior Q value function of the base station according to a formula: the Q matrix update formula is:

   

   wherein r(s)_t,a_t) For agent in state s_tWhile performing action a_tThe obtained reward value, alpha represents the learning rate and 0<α<Y 1, y represents discount factor and 0 ≦ y<1, adjusting the updating of a Q matrix under the synergistic action of a learning rate and a discount factor so as to influence the learning performance of a Q algorithm, wherein alpha is 0.01, and gamma is 0.8;

   s6: t ← t +1, go to step S2.

Images