CN115915454A - SWIPT-assisted downlink resource allocation method and device - Google Patents

Abstract

The application discloses a method and a device for downlink resource allocation assisted by SWIPT. Wherein, the method comprises the following steps: obtaining a state observation value of a current environment state of a communication link between tolerant machine equipment user equipment and a machine equipment gateway; based on the state observation value of the current environment state, selecting a resource allocation strategy for the tolerant machine user equipment by using a resource allocation model constructed based on a neural network; allocating downlink resources to the tolerant machine user equipment based on the selected resource allocation strategy; wherein the tolerant machine user device requires a machine user device that completes a task of transmission of the periodically generated payload. The method and the device solve the technical problems that the machine user equipment with the energy limitation characteristic can generate overhigh energy consumption and uneven resource distribution of various internet of things equipment when a communication link is established, and particularly the machine user equipment with low QoS (quality of service) requirements can obtain poor resource distribution and therefore the performance is poor.

Description

SWIPT-assisted downlink resource allocation method and device

Technical Field

The present application relates to the field of communications, and in particular, to a SWIPT-assisted downlink resource allocation method and device.

Background Art

The rapid development of Internet of Things technology has spawned data-driven fields such as interconnected industries, intelligent transportation systems and smart cities. Driven by these fields, human society is gradually becoming digital.

Machine-to-machine communication (M2M) is a major factor in the development of digitalization due to its near-instant wireless connection without direct human intervention. With the exponential growth in the number of connected machines and devices, M2M communication will provide new possibilities for citizen services in scenarios such as home, entertainment, medical care and work, and is expected to become a key driver for realizing the beautiful vision of the intelligent connection of all things.

Different from traditional human-to-human communication (H2H), M2M communication services have their own unique characteristics in terms of network energy efficiency, data transmission volume, and high-reliability transmission in some mission-critical services. Although low-power wide-area networks have been developed in IoT technology specifically for M2M communication, which can provide M2M devices with extremely low power consumption and long-distance transmission, the network can only provide a very low transmission rate and is difficult to meet the QoS requirements of most critical M2M services. Therefore, cellular networks are considered to be the key factor in deploying M2M communications because of their larger bandwidth and stronger network scalability, which can meet the access of various services and provide high data transmission rates. At the same time, cellular networks themselves are too energy-consuming and costly for many M2M communication applications.

In order to mitigate the negative impact of cellular networks on M2M communications, SWIPT (Wireless Power In-Person Transmission) is a promising technology that can significantly improve the energy efficiency of M2M devices in H2H/M2M coexisting cellular networks. Although traditional energy harvesting technologies allow mobile devices or base stations to extract energy from natural environments such as wind or sunlight, the efficiency of such technologies is severely constrained by geographical location and weather conditions. SWIPT, as a radio frequency-based energy harvesting technology, enables application devices to obtain relatively controllable and stable energy. Generally, this technology can convert interference signals into electrical energy, that is, a strong interference environment will reduce system throughput while also bringing more energy collection. In addition, SWIPT-assisted H2H/M2M coexisting cellular networks face many challenges in meeting users' diverse quality of service (QoS) requirements, such as intra-layer and inter-layer interference control, spectrum sub-band selection, and the trade-off between system information transmission rate and energy collection.

To address the above-mentioned problems, no effective solution has been proposed yet.

Summary of the invention

The embodiments of the present application provide a SWIPT-assisted downlink resource allocation method and apparatus to at least solve the technical problem that a machine user equipment obtains poor resource allocation and thus has poor performance.

According to one aspect of an embodiment of the present application, a SWIPT-assisted downlink resource allocation method is provided, including: obtaining a state observation value of a current environmental state of a communication link between a tolerant machine user device and a machine device gateway; based on the state observation value of the current environmental state, selecting a resource allocation strategy for the tolerant machine user device using a resource allocation model constructed based on a neural network; based on the selected resource allocation strategy, allocating downlink resources to the tolerant machine user device; wherein the tolerant machine user device is a machine user device that needs to complete a transmission task of a periodically generated load.

According to another aspect of an embodiment of the present application, a SWIPT-assisted downlink resource allocation device is also provided, including: an acquisition module, configured to obtain a state observation value of a current environmental state of a communication link between a tolerant machine user device and a machine device gateway; a selection module, configured to select a resource allocation strategy for the tolerant machine user device based on the state observation value of the current environmental state and a resource allocation model constructed based on a neural network; an allocation module, based on the selected resource allocation strategy, allocates downlink resources to the tolerant machine user device; wherein the machine user device needs to complete the transmission task of a periodically generated load.

In an embodiment of the present application, based on the state observation value of the current environmental state, a resource allocation strategy is selected for a tolerant machine user device using a resource allocation model built based on a neural network, thereby solving the technical problem that the machine user device obtains poor resource allocation and thus has poor performance.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

FIG1 is a flow chart of a SWIPT-assisted downlink resource allocation method according to an embodiment of the present application;

FIG2 is a flow chart of a method for constructing a downlink resource allocation model according to an embodiment of the present application;

FIG3 is a flow chart of another SWIPT-assisted downlink resource allocation method according to an embodiment of the present application;

FIG4 is a flowchart of another SWIPT-assisted downlink resource allocation method according to an embodiment of the present application;

FIG5 is a schematic structural diagram of a SWIPT-assisted downlink resource allocation device according to an embodiment of the present application;

FIG6 is a schematic structural diagram of a SWIPT-assisted downlink resource allocation system according to an embodiment of the present application;

FIG7 is a schematic diagram showing a comparison between the change in the number of M2M devices and the total energy efficiency of the M2M devices according to an embodiment of the present application;

FIG8 is a schematic diagram showing a comparison between the change in the number of tolerant M2M devices and the satisfaction rate of H2H user QoS requirements according to an embodiment of the present application;

9 is a schematic diagram showing a comparison between the change in the number of tolerant M2M devices and the probability of critical M2M link interruption according to an embodiment of the present application;

FIG. 10 is a schematic diagram showing a comparison between the number of tolerant M2M devices and the load transmission completion rate according to an embodiment of the present application.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

Example 1

According to an embodiment of the present application, a SWIPT-assisted downlink resource allocation method is provided, as shown in FIG1 , the method comprising:

Step S102: obtaining a state observation value of a current environment state of a communication link between a tolerant machine user device and a machine device gateway.

Before obtaining the state observation value of the current environmental state of the communication link between the tolerant machine user device and the machine device gateway, it is necessary to divide the machine user device into tolerant machine user devices and critical machine user devices based on the service type of the machine user device; among which, the critical machine user device is a machine user device with a transmission reliability requirement higher than a preset requirement threshold, and the tolerant machine user device is a machine user device that needs to complete the transmission task of the periodically generated load.

In addition to considering the existing conventional H2H (Human-to-Huma) user devices (also called human user devices, or H2H users), this embodiment further subdivides machine user devices (Machine-Type Communication Devices, MTCD) into tolerant machine user devices (also called tolerant M2M devices, tolerant M2M users) and critical machine user devices (also called critical M2M devices, critical M2M users), so that different resource allocation strategies can be selected according to the service characteristics of different machine user devices, thereby allocating resources to machine user devices more reasonably.

In one example, the machine user device may be provided with a Simultaneous Wireless Information and Power Transfer (SWIPT) to enable the machine user device to obtain energy from the radio frequency environment and decode information at the same time. By setting SWIPT, the MTCD can obtain energy and decode information at the same time.

Step S104, based on the state observation value of the current environment state, using the resource allocation model constructed based on the neural network, select a resource allocation strategy for the tolerant machine user device.

In one example, as shown in FIG. 2 , a resource allocation model may be constructed based on the following method:

Step S1042, constructing a state function.

The state function is a collection of state observations. For example, the state function is constructed based on the channel gain information of the communication link on each spectrum sub-band, the interference power of the communication link on each spectrum sub-band, the load remaining and transmission time remaining of the communication link, the current number of iterations, and the greed factor representing the current environment exploration rate of the communication link.

Step S1044, constructing an action function.

The action function is a collection of downlink spectrum resources, transmit power level and power split ratio.

Step S1046, construct a reward function.

The reward function is constructed based on the balance between resource allocation optimization objectives and QoS constraints.

In some examples, the reward function can be constructed as follows.

First, based on the load remaining amount and transmission time remaining of the tolerant machine user device, the penalty item of the reward function provided by the tolerant machine user device is determined; then, based on the interruption probability of the critical machine user device, the penalty item of the reward function provided by the critical machine user device is determined; then, based on the signal to interference and noise ratio of the human user device, the penalty item of the reward function of the human user device is determined; finally, a reward function is constructed based on the total energy efficiency value of the communication link of all machine user devices, the penalty item of the reward function provided by the tolerant machine user device, the penalty item of the reward function provided by the critical machine user device, and the penalty item of the reward function of the human user device.

In this embodiment, for QoS constraints, the threshold of the signal-to-interference-and-noise ratio of the H2H user equipment, the interruption probability of the critical M2M communication link, and the effective load transmission probability of the M2M communication link are explicitly modeled and solved, and the corresponding penalty term of the reward function is obtained, so that the reward value of the reward function is more accurate, thereby improving the rationality of resource allocation.

In some other examples, the reward function can be constructed as follows.

First, set the Quality of Service (QoS) constraint. For example, set the QoS constraint of human user equipment to the signal to interference plus noise ratio SINR greater than the set minimum threshold; set the QoS constraint of tolerant machine user equipment to the transmission success rate of a load of a preset size V within the time constraint T higher than the set success rate threshold; set the QoS constraint of critical machine user equipment to the interruption probability not higher than the set interruption threshold.

Next, the total energy efficiency value EE achieved by the tolerant machine user equipment and the critical machine user equipment is used as the resource allocation optimization target. This embodiment aims to maximize the EE of the M2M communication link and allocates the joint spectrum, transmit power and power split PS ratio under the consideration of user QoS constraints.

Finally, the reward function is constructed based on the resource allocation optimization goal and QoS constraints.

Step S1048, constructing an experience replay pool, wherein the experience replay pool is used to store data and provide data for training the neural network.

An experience replay pool is constructed to store training data for training the resource allocation model, wherein the training data includes state observation values, reward values and selected resource allocation strategies at the current moment and the next moment; wherein the neural network includes: a training network and a target network, wherein the training network uses data randomly extracted from the experience replay pool in each round of iteration and is trained by stochastic gradient descent; the target network is a fixed neural network, and its network parameters are updated at regular intervals to the training network parameters at the current moment.

Step S1049, constructing and training a resource allocation model based on the state function, action function, reward function and experience replay pool.

Step S106: allocating downlink resources to the tolerant machine user equipment based on the selected resource allocation strategy.

After obtaining the state observation value of the current environment state, or after allocating downlink resources, data related to the allocated resources can also be put into the experience replay pool corresponding to each communication link to be used as training data.

For example, the total energy efficiency value of all machine user devices and the QoS level of each machine user device and human user device at the current moment are calculated, and the total energy efficiency value, QoS level, the interference power received by the communication link on each spectrum sub-band, the selected resource allocation strategy, the state observation value, the reward output by the reward function, and the prior information received by the machine device gateway are put into the experience replay pool of each communication link to be used as training data for training the resource allocation model.

This embodiment solves the energy resource allocation problem in SWIPT-assisted H2H/M2M (Machine-to-Machine) coexistence cellular networks. By allocating appropriate resources (i.e., spectrum, power, and power splitting (PS) ratio) to tolerant machine user equipment, high energy efficiency (EE) is achieved and QoS requirements are guaranteed.

This embodiment provides a stable training process based on the state of behavior tracking, and the shared reward function enables each agent to work together in a distributed manner. In addition, this embodiment provides a mathematically precise expression of the QoS constraint, thereby reducing the computational complexity of the reward function.

In addition, in order to support different QoS requirements, this embodiment models the EE optimization problem of multiple M2M communication links as a multi-agent problem, where multiple tolerant M2M communication links share the spectrum occupied by critical M2M communication links and H2H communication links. In addition, this embodiment enables the agent to adapt to the dynamic network environment by designing the state function, action function and reward function, thereby achieving the effect of target optimization.

Example 2

According to an embodiment of the present application, a SWIPT-assisted QoS-based downlink resource allocation method is also provided, as shown in FIG3 , the method comprising:

Step S302: Initialize the resource allocation system.

Based on the pre-assigned spectrum sub-bands and transmit power, the channel conditions of the spectrum sub-bands occupied by H2H users and critical M2M users and prior information such as SINR or outage probability are obtained. More specifically, initialization refers to determining the initial state of each tolerant M2M link at time t

Step S304, building a training neural network and a target neural network for each agent.

Each tolerant M2M link is regarded as an agent, and a training neural network Q(s, a; ω) and a target neural network ^Q- (s, a; ^ω- ) are built for each agent, where ω and ^ω- represent the parameters of the training network and the target network, respectively. The input layer of the network is s, that is, the state observation value of each tolerant link, and the output layer corresponds to the estimated reward value corresponding to each action selection a, expressed as Q value.

An experience replay pool D is established for each agent to store the state observation values S ^t and S ^{t+1 of its own link at the current time t and the next time t+1} , the action selection A ^t at the current time t, and the reward value R ^t+1 received at the next time. The data stored in D at time t is represented by the topology (S ^t , A ^t , R ^t+1 , S ^t+1 ).

Step S306, designing a state function, an action function and a reward function for the agent.

Each agent builds a link communication energy efficiency optimization model based on prior information and action selection according to local state observations, including: setting constraints on transmit power, power split ratio, and QoS constraints.

Among them, QoS constraints are modeled as follows according to the actual characteristics of the services: H2H services are usually traffic based on voice and Internet communications, which places high demands on data transmission rates. Therefore, the QoS constraints of H2H services are modeled as SINR values greater than the set minimum threshold; most M2M services are event-driven, and their traffic is usually generated periodically at different frequencies according to the needs of the M2M services themselves. Therefore, the QoS constraints of tolerant M2M services are modeled as the transmission success rate of data packets of size V within the time constraint T; in addition, for critical M2M services, they often have strict delay and transmission reliability requirements. Therefore, the QoS constraints of critical M2M services are modeled as the interruption probability must not be higher than the set interruption threshold.

Step S308: training the training network Q and allocating network resources.

Based on the link communication energy efficiency optimization model, the tolerant M2M link continuously randomly extracts data from a large amount of data stored in the experience replay pool to train the training network Q, and reduces the error between the training network and the target network parameters by stochastic gradient descent. Finally, after the resource allocation model training converges, a resource allocation plan that meets the constraints and QoS constraints can be made.

In addition, in order to stabilize the training environment, this implementation adopts a low-dimensional mapping high-dimensional behavior tracking method, so that each tolerant M2M link can know the current state of the learned experience when facing the training data randomly selected from the experience replay pool. And, in order to improve the overall energy efficiency performance, this embodiment shares the reward function to encourage each tolerant M2M link to explore the environment in a cooperative manner and make intelligent decisions that maximize the overall performance.

In one embodiment, the specific process of resource allocation may include: the tolerant M2M link selects the spectrum sub-band with the best channel condition for multiplexing by the machine device gateway according to the received prior information, and allocates appropriate transmission power and power split ratio to the tolerant machine devices in the link. At each moment (1ms), the spectrum sub-band channel condition in the network will change due to channel fading. In addition, the resource allocation selection made by each tolerant M2M link at each moment will also affect the QoS indicators of various types of users in the network. Tolerant M2M judges the merits of the resource allocation scheme at this moment based on these changed QoS indicators, namely, the SINR of H2H users, the interruption probability of critical M2M users, the remaining load and transmission time of tolerant M2M users, and the achievable energy efficiency of all M2M users, and stores the current channel conditions, QoS indicators, interference, resource selection scheme and achievable energy efficiency in the experience replay pool as training data for training network Q in each tolerant M2M link. As the environmental data continues to change, the training data samples continue to increase. By randomly extracting data from the experience replay pool for training, data correlation can be effectively eliminated.

For the spectrum subband selection, transmission power and power split ratio allocation tasks of tolerant M2M links, a distributed implementation scheme is adopted. At each time t, each tolerant M2M link selects the spectrum subband with the optimal channel condition and the appropriate transmission power and power split ratio according to the above-mentioned prior information and its own training network, and communicates with the resource allocation scheme, and then obtains the overall energy efficiency of all M2M devices and the QoS indicators of other users in the network, which is called the reward function. All tolerant M2M links in the network share this same reward function, so the resource allocation method proposed in this application encourages cooperation between links to achieve the goal of maximizing the overall energy efficiency of M2M.

The resource allocation task further includes: at each time t, according to the prior information received by the machine equipment gateway MTCG m in the cluster, the available environment data set of the tolerant M2M link m, n is updated, and its state set is designed as follows:

包含了链路m，n在每个频谱子带上的信道增益信息，

描述了链路m，n每个频谱子带上受到的干扰功率大小，V_m，n，T_m，n分别表示链路m，n的载荷剩余和传输时间剩余，{QoS_h}_h∈H，{QoS_s}_s∈S为二元变量，分别表示H2H用户和关键型M2M用户的QoS指标，即所有H2H用户的QoS约束得到满足，则{QoS_h}_h∈H＝1，否则为0，{QoS_s}_s∈S以相同的二元变量表示所有关键型M2M链路的QoS约束是否得到满足。in,

Contains the channel gain information of link m, n in each spectrum subband,

It describes the interference power on each spectrum sub-band of link m, n. V _m,n , T _m,n represent the load surplus and transmission time surplus of link m, n respectively. {QoS _h } _h∈H , {QoS _s } _s∈S are binary variables, representing the QoS indicators of H2H users and critical M2M users respectively. That is, if the QoS constraints of all H2H users are met, {QoS _h } _h∈H ＝1, otherwise it is 0. {QoS _s } _s∈S uses the same binary variable to indicate whether the QoS constraints of all critical M2M links are met.

In addition, e represents the current number of iterations, ∈ is the greed factor, which indicates the current environment exploration rate of link m, n. The embodiment of the present application adds these two items to the state set to track the environment state at the iteration number e and the exploration probability ∈ during the training process: during the training process, the resource allocation strategy of each tolerant M2M link will change with the change of the resource allocation strategy of other links. Therefore, from the perspective of a single link, the network environment is changing all the time. Moreover, since the experience extracted from the experience replay pool is random, the training data extracted at the current moment no longer reflects the network environment at this time, but is most likely outdated data, which makes it difficult for each link to know the state of the experience and the strategy selection of each link when facing the training data. The value function that truly represents the strategy selection of each link is a high-dimensional neural network parameter, which cannot be used as experience data for each link to learn. Therefore, a design is made to add the iteration number e and the greed factor ∈ to the experience data to map the high-dimensional neural network parameters, so that the tolerant M2M link can track the environment state when facing the selected experience data, thereby achieving the purpose of stable training.

According to the above environmental information, the tolerant M2M link m, n uses the greedy strategy at time t: randomly select the spectrum subband, transmit power and power split ratio with probability ∈ or select the optimal action selection strategy given by the current training network with probability 1-∈. The action (resource) set is defined as:

表示频谱子带选择，

分别代表了发射功率和功率分流比选择，

代表最大发射功率，L和Z分别代表功率和功率分流比的分割等级。in

represents the spectrum subband selection,

Represents the transmit power and power split ratio selection respectively,

represents the maximum transmit power, L and Z represent the division level of power and power split ratio respectively.

In this embodiment, it is assumed that the spectrum subband set available in the network is indexed by K=H∪S, where H={1, 2...H} represents HUE, and S={1, 2...S} represents critical machine user device (CMTCD). In addition, M={1, 2...M} represents MTCG, and each MTCG-controlled tolerant machine user device (TMTCD) is represented by N={1, 2...N}, where all TMTCDs and CMTCDs are equipped with SWIPT technology, represented by set D, that is, D=S∪N.

建立通信链路，根据频谱子带选择，高QoS需求用户的SINR分为如下两种计算情况：Action selection made by tolerant M2M link m, n at time t

When establishing a communication link, based on the spectrum sub-band selection, the SINR of users with high QoS requirements is calculated in the following two ways:

表示H2H用户或关键型M2M用户于频谱子带k上受到的干扰，

为复用第k个频谱子带时受到的跨集群干扰和同集群干扰，

为表示频谱子带占用情况的二元变量，第二项σ²表示加性高斯白噪声功率。P_BS，h和P_m，s分别代表由基站和机器设备网关分配给H2H用户和关键型M2M用户的发射功率，ρ_m，s表示机器设备网关分配给关键型M2M用户的功率分流比，P_BS，h表示基站分配给H2H用户的发射功率，

表示基站与H2H用户间的信道增益，

表示机器设备网关与H2H用户间的信道增益，

表示同一集群内机器设备网关与关键型M2M用户间的信道增益。在计算得到H2H用户和关键型M2M用户的SINR后，利用如下公式：In the above formula, the first term in the denominator is

represents the interference received by H2H users or critical M2M users on spectrum sub-band k,

is the cross-cluster interference and intra-cluster interference when reusing the kth spectrum sub-band,

is a binary variable representing the occupancy of the spectrum subband, and the second term σ ² represents the additive white Gaussian noise power. _{P BS,h} and P _m,s represent the transmit power allocated by the base station and the machine device gateway to the H2H user and the critical M2M user, respectively. ρ _m,s represents the power split ratio allocated by the machine device gateway to the critical M2M user. P _BS,h represents the transmit power allocated by the base station to the H2H user.

Indicates the channel gain between the base station and the H2H user,

Indicates the channel gain between the machine device gateway and the H2H user.

It represents the channel gain between the machine device gateway and the key M2M user in the same cluster. After calculating the SINR of H2H users and key M2M users, the following formula is used:

和

分别表示H2H业务和关键型M2M业务的SINR阈值，

指可容忍的最大中断概率。若满足H2H业务的QoS约束，则环境状态信息{QoS_h}_h∈H＝1，否则{QoS_h}_h∈H＝0，相应地，{QoS_h}_s∈S以相同的指示符表示所有关键型M2M用户的QoS满意度。To determine whether the QoS requirements of H2H users and critical M2M users are guaranteed.

and

Respectively represent the SINR thresholds for H2H services and critical M2M services,

Refers to the maximum tolerable interruption probability. If the QoS constraint of the H2H service is met, the environment status information {QoS _h } _h∈H ＝1, otherwise {QoS _h } _h∈H ＝0. Correspondingly, {QoS _h } _s∈S represents the QoS satisfaction of all critical M2M users with the same indicator.

For the tolerant M2M link itself, its SINR value is expressed by the following formula:

是第m，n条容忍型M2M链路受到的干扰，根据复用的频谱子带的占有者不同区分为如下两种情况：in,

The interference to the mth and nth tolerant M2M links is divided into the following two cases according to the occupants of the multiplexed spectrum sub-bands:

表示容忍型M2M链路m，n于频谱子带k上的信道增益，

表示频谱子带分配情况指示符，

表示第m′个机器设备网关在频谱子带k上的发射功率，

表示第m′个机器设备网关和容忍型M2M设备n于频谱子带k上的信道增益，

表示频谱子带分配情况指示符，

表示容忍型M2M链路m，n于频谱子带k上进行通信，

表示容忍型M2M链路m，n未于频谱子带k上进行通信，

表示机器设备网关m分配给容忍型M2M用户n的发射功率，M表示机器设备网关总数，N表示一个机器设备网关管控范围内的容忍型M2M设备总数。Where Pm _,s represents the transmission power allocated by the machine equipment gateway to the critical M2M user.

represents the channel gain of tolerant M2M link m, n on spectrum subband k,

Indicates the spectrum sub-band allocation indicator.

represents the transmission power of the m′th machine device gateway on spectrum subband k,

represents the channel gain of the m′th M2M gateway and tolerant M2M device n on spectrum subband k,

Indicates the spectrum sub-band allocation indicator.

represents a tolerant M2M link m, n communicating on spectrum subband k,

Indicates that the tolerant M2M link m, n does not communicate on the spectrum subband k,

It represents the transmission power allocated to the tolerant M2M user n by the machine device gateway m, M represents the total number of machine device gateways, and N represents the total number of tolerant M2M devices within the control range of a machine device gateway.

Considering that the information that the tolerant M2M service needs to process is often data packets generated periodically at different frequencies according to its own service characteristics, its QoS is modeled as the success rate of completing data packet transmission within the time budget T:

表示链路m，n在不同时刻t内，于频谱子带k上的信息传输速率，B是每个频谱子带的带宽，上式中V代表周期性产生的M2M载荷，单位为比特，Δ_T为信道相干时间。in,

represents the information transmission rate of link m, n on spectrum subband k at different times t, B is the bandwidth of each spectrum subband, V represents the periodically generated M2M load in bits, and Δ _T is the channel coherence time.

The EE of all SWIPT-assisted M2M devices can be expressed as the ratio of spectrum efficiency to energy consumption, and all M2M devices are represented by D = S ∪ N:

in

Represents the achievable spectrum efficiency of all M2M devices;

represents the energy consumption of all M2M devices. Where _PC represents the power consumption of all M2M links.

represents the energy harvested by the M2M device, where θ∈[0,1] is the energy conversion efficiency.

So far, the M2M link energy efficiency optimization model of this application is summarized as follows:

和

分别代表了功率分流比发射功率分配和频谱子带分配策略，条件(1a)(1b)(1c)分别代表了H2H用户、关键型M2M用户和容忍型M2M用户的QoS约束，条件(1d)规定了应用SWIPT的M2M设备的功率分流比不大于1，(1e)中的

是二元变量，容忍型M2M链路m，n被分配频谱子带k表示为1，否则为0，(1f)中的

规定了容忍型M2M链路的发射功率上界，(1g)用于约束每条容忍型M2M链路最多选择一个频谱子带进行通信。in

and

represent the power split ratio transmit power allocation and spectrum sub-band allocation strategy respectively. Conditions (1a), (1b) and (1c) represent the QoS constraints of H2H users, critical M2M users and tolerant M2M users respectively. Condition (1d) stipulates that the power split ratio of the M2M device applying SWIPT is not greater than 1. Condition (1e)

is a binary variable, and the tolerant M2M link m,n is assigned spectrum subband k, which is 1, otherwise it is 0.

The upper bound of the transmission power of the tolerant M2M link is specified, and (1g) is used to constrain each tolerant M2M link to select at most one spectrum subband for communication.

For each communication and interference link channel gain model (where g represents the channel gain), it is summarized as follows:

计算同一集群内，机器设备网关m与容忍型机器设备n间组成的M2M通信链路m，n于频谱子带k上的信道增益。其中X_m，nβ_m，n表示路径损耗和阴影衰落，称为大尺度衰落，这两项与频率无关且在较长时间内保持不变，

表示与频率相关的快衰落。use

Calculate the channel gain of the M2M communication link m,n between the machine device gateway m and the tolerant machine device n in the same cluster on the spectrum subband k. Where X _m,n β _m,n represents the path loss and shadow fading, called large-scale fading, which are independent of frequency and remain unchanged over a long period of time.

Indicates frequency-dependent fast fading.

计算同一集群内，机器设备网关m与关键型机器设备s间组成的M2M通信链路m，s于频谱子带k上的信道增益。use

Calculate the channel gain of the M2M communication link m,s formed between the machine device gateway m and the key machine device s in the same cluster on the spectrum subband k.

计算基站BS与人类型用户设备h间组成的H2H通信链路BS，h于频谱子带k上的信道增益。use

Calculate the channel gain of the H2H communication link BS,h formed between the base station BS and the human-type user equipment h on the spectrum sub-band k.

计算基站BS与容忍型M2M链路m，n于频谱子带k上的干扰信道增益。use

Calculate the interference channel gain between the base station BS and the tolerant M2M link m, n on the spectrum subband k.

计算不同集群间，机器设备网关m′与容忍型M2M链路m，n于频谱子带k上的干扰信道增益。use

Calculate the interference channel gain between the machine device gateway m′ and the tolerant M2M link m,n on the spectrum subband k between different clusters.

计算不同集群间，机器设备网关m′与关键型M2M链路m，s于频谱子带k上的干扰信道增益。use

Calculate the interference channel gain between the machine device gateway m′ and the critical M2M link m,s on the spectrum subband k between different clusters.

计算机器设备网关m与人类型用户设备h于频谱子带k上的干扰信道增益。use

The interference channel gains between the machine device gateway m and the human type user equipment h on the spectrum sub-band k are calculated.

In order to enable multiple tolerant M2M links to move toward the goal of maximizing overall energy efficiency during training while ensuring QoS constraints for various types of users, this application adds overall EE and QoS constraints to the design of the reward function.

Obviously, at each time t, the SINR value of the H2H user can be easily obtained from the power and interference received by the user, while the QoS constraint of the M2M user is a probability value. At each time, each agent needs to randomly generate a large number of simulated channels to obtain the required probability value, thereby consuming a large amount of computing resources and slowing down the convergence speed of the algorithm. In order to solve this problem, this application converts such QoS constraints into precise explicit expressions.

First, use U _n to represent the remaining transmission time of the tolerant M2M user when the load is not transmitted. After the transmission is completed, U _n is set to a constant value. Therefore, at time t, the reward and penalty related to the tolerant M2M user are set as:

Through such transformation, each agent can consider the impact of both transmission time surplus and load transmission rate during training.

Secondly, the present application uses theoretical analysis and mathematical transformation to obtain the precise value of the interruption probability of the critical M2M device. When the sth critical M2M device multiplexes and shares the kth spectrum subband with tolerant M2M devices in different clusters, the interruption probability can be replaced as follows:

的独立指数分布随机变量，可得公式：According to the lemma: Assume that z ₁ , ..., z _n are

Independent exponentially distributed random variables, we can get the formula:

Where c is a positive constant. Therefore, according to the Rayleigh fading characteristics and the above formula, the outage probability expression can be rewritten as:

根据所受干扰源的不同可以区分为m或m′，分别表示集群间干扰与集群内干扰。此外，

也可以同时包含m和m′，表示此时受到的干扰既包含集群间干扰也包括集群内干扰。值得注意的是，当受到的干扰仅为集群间干扰时，上式的累乘项

仅保留Π_n项。相应地，

表示为第s个CMTCD于频谱子带k上所受到的干扰，表示为in

According to the different interference sources, it can be divided into m or m', which represent inter-cluster interference and intra-cluster interference respectively.

It can also include both m and m′, indicating that the interference received at this time includes both inter-cluster interference and intra-cluster interference. It is worth noting that when the interference received is only inter-cluster interference, the cumulative term in the above formula is

Only the Π _n term is retained. Accordingly,

It is represented by the interference received by the s-th CMTCD on the spectrum sub-band k, which is expressed as

同时包含m和m′时，中断概率的计算需要连续累乘上式中的所有项，从而产生较大的计算资源开销。In addition, when

When both m and m′ are included, the calculation of the outage probability requires continuous multiplication of all terms in the above formula, which results in a large computational resource overhead.

In order to further accelerate the convergence of the algorithm, the above formula is converted into the following form:

来表示在时隙t内关键型M2M设备的QoS满意度，表示如下：The above formula can be used to obtain the accurate interruption probability value of each critical M2M device with less computing resource consumption. In order to enable each agent to further distinguish the quality of the selected resource allocation strategy during the training process, this application further sets

To express the QoS satisfaction of the critical M2M device in time slot t, it is expressed as follows:

来表示H时隙t内2H用户的QoS满意度：Where ps is the interruption probability value obtained by using the interruption probability expression derived above. Similarly,

To express the QoS satisfaction of 2H users in H time slot t:

In summary, this application sets the global reward function as:

Where μ is a positive real number to balance rewards and penalties. By making all agents share the same reward function, the agents can cooperate to explore the location environment and learn the resource allocation strategy that maximizes the overall energy efficiency. At the same time, each penalty term can ensure that each agent considers the QoS constraints of various users when allocating resources.

After each round of iteration, each agent randomly extracts small batches of data from the experience replay pool to train the training network Q, and updates the training network parameters ω by stochastic gradient descent to approach the target network parameters ω ^- , and passes its own network parameters to the target network after one iteration.

The target network is represented as:

Among them, 0≤γ≤1 is the discount factor, and its value determines the importance of the current reward and punishment value in the iteration process. The smaller it is, the more important the current reward and punishment value is, and vice versa. ^St+1 represents the state observation value at the next moment, a′ represents the optimal action selection made under the state ^St+1 , and ω ^- represents the target network parameter.

In this embodiment, the loss function is expressed as:

L(θ)=E[(Q ^- -Q(S ^t , A ^t ; ω)) ² ]

Among them, L(θ) represents the loss function, E represents the mathematical expectation, Q represents the training network value function, ^St represents the state observation value at time t, A ^t represents the action selection under the state ^St , and ω represents the parameters of the training network.

In the embodiment of the present application, the SWIPT technology is equipped for the M2M device in the H2H/M2M coexistence cellular network. In the complex interference environment caused by spectrum sharing, high EE performance is achieved and the QoS requirements of all types of users are ensured by allocating spectrum, transmission power and power split ratio resources to tolerant M2M links.

In addition, in an embodiment of the present application, in addition to the traditional H2H services, the SWIPT-assisted H2H/M2M coexistence cellular network also divides the M2M service types into critical services and tolerant services, wherein M2M devices exist in a cluster form and establish a communication link with a local machine device gateway, and all M2M devices are equipped with SWIPT technology, so that they can simultaneously perform energy collection and information decoding; H2H users and critical M2M users in the network are pre-allocated OFDM spectrum sub-bands, and the remaining tolerant M2M users reuse the same spectrum resources.

In the resource allocation method proposed in this application, the energy efficiency optimization problem is modeled as a multi-agent problem. Each tolerant M2M link is taken as an agent, and a set of systematic state, action, and reward function design schemes are proposed. Through the designed state set and reward function based on behavior tracking, each agent can more efficiently select appropriate spectrum sub-bands, transmission power, and power split ratio resources from the action set in a distributed and cooperative manner during the interaction with the unknown network environment, so as to achieve the purpose of optimizing the overall M2M energy efficiency. In addition, this application further reduces the computational complexity in the model training process by expressing the QoS requirements (QoS constraints) of each type of user with precise mathematical formulas.

Example 3

According to an embodiment of the present application, another SWIPT-assisted QoS-based downlink resource allocation method is also provided, as shown in FIG4 , the method comprising:

Step S402, initialization.

Initialize the resource allocation system, and obtain the channel conditions of the spectrum subbands occupied by the H2H link and the critical M2M link and prior information such as SINR or outage probability based on the pre-allocated spectrum subbands and transmit power.

Step S404, building a neural network and establishing an experience replay pool for each agent.

Each tolerant M2M link is regarded as an agent and a training neural network and a target neural network are built, and an experience replay pool is established for each agent.

In the embodiment of the present application, the resource allocation problem is first modeled as a multi-agent intelligent decision-making problem, where each tolerant M2M link is regarded as an agent for interacting with the unknown network environment and optimizing the resource allocation strategy. Then, a training network, a target network and an experience replay pool are constructed for each agent using a resource allocation method, where the experience replay pool stores data for training the training network.

Step S406, designing a state function, an action function and a reward function for the agent.

Each agent builds a link communication energy efficiency optimization model based on the prior information and the action selection made according to the local state observation value, and stores the M2M system energy efficiency, various QoS indicators and the interference data of the agent itself returned by the link energy efficiency optimization model into the experience replay pool.

Designing the state function, action function, and reward function for the agent involves the following steps:

S3-1, a low-dimensional mapping high-dimensional trajectory tracking method is designed in the state set: the number of iterations and the greedy factor are added to the state set, so that the agent can track the environment state at the iteration number e and the exploration probability ∈ during the training process when extracting training data. This solves the problem of non-stationarity of the training environment caused by the multi-agent setting.

S3-2, the agent establishes a link communication energy efficiency optimization model based on the state observation value and the selected action, specifically including: formulating the constraints of the transmission power and power split ratio and the QoS requirements, wherein the QoS requirements are modeled according to the actual characteristics of the service as follows: H2H services are usually based on voice and Internet communication traffic, which places high demands on data transmission rate, so the QoS requirements of H2H services are modeled as SINR values greater than the set minimum threshold; most M2M services are event-driven, and their traffic is usually generated periodically at different frequencies according to the needs of the M2M services themselves, so the QoS requirements of tolerant M2M services are modeled as the transmission success rate of data packets of size V within the time constraint T; in addition, for critical M2M services, they often have strict delay and transmission reliability requirements, so the QoS requirements of critical M2M services are modeled as the interruption probability must not be higher than the set threshold.

S3-3, the reward function is designed as a combination of the main reward term and the penalty term, and all agents share the same reward function, which encourages the agents to cooperate in exploring the optimal resource allocation strategy while taking into account the QoS requirements of various users. In addition, in order to reduce the consumption of computing resources and speed up the convergence speed, various QoS requirements are accurately mathematically expressed in the penalty terms of the reward function;

S3-4: Calculate the total EE of the M2M devices in the current time slot network and the QoS level of each type of user through the link optimization model, and store the calculated EE, QoS level, interference suffered by the tolerant M2M link, the selected action, the state observation value, the reward obtained, and the prior information received by the machine device gateway into the experience replay pool as training data for training the network;

Step S408: training a resource allocation model and allocating resources.

Based on the link communication energy efficiency optimization model and the experience replay pool, the agent continuously randomly extracts data from the large amount of data stored in the experience replay pool to train the training network Q, and reduces the error between the training network and the target network parameters in a stochastic gradient descent manner. Finally, after the model training converges, a resource allocation plan that meets the constraints and QoS requirements can be made.

By randomly extracting batches of data from the experience replay pool to train the training network, it is possible to learn current and past experiences at the same time, and effectively eliminate data correlation. By asynchronously updating the target network parameters, the target network parameters remain unchanged for a period of time, making the algorithm update more stable.

SWIPT technology has two receiver designs, namely "time switching" and "power splitting", where "time switching" switches between the duration of information decoding and energy collection, while "power splitting" uses the power split ratio to split the received energy into the information decoding part and the energy collection part. This embodiment adopts the "power splitting" design, so the power split ratio allocation is introduced in the action set design of the resource allocation method. In other embodiments, "time switching" can also be used to replace the power split ratio allocation with time allocation.

In the H2H/M2M coexistence cellular network used by the prior art, due to the complex network structure of the H2H/M2M coexistence cellular network and the changeable resource allocation tasks, the traditional resource allocation scheme has poor performance, and it ignores the rich service types of machine user equipment and the interference caused by spectrum sharing; and SWIPT is not used in the network model considering EH to obtain stable and reliable energy. In the practical application of the prior art, it does not accurately mathematically model the QoS requirements of each type of user, which often leads to poor performance of its user equipment as a low QoS demand side; in addition, the proposed traditional algorithm has high computational complexity and poor effect when solving optimization problems with nonlinear constraints, while the centralized training of the proposed intelligent algorithm has a slow convergence speed.

Compared with the existing intelligent resource allocation method, the embodiment of the present application introduces SWIPT-assisted M2M devices into the network structure to obtain stable and reliable energy from the radio frequency environment.

In addition, in the implementation of the resource allocation method, a multi-agent cooperation and distributed execution method is selected. Since the resource allocation scheme of the multi-agent distributed execution can take into account the situation of each agent and make the most suitable resource allocation strategy, and the resource allocation strategy made by the single-agent centralized execution tends to be applicable to all agents in the network, this embodiment greatly reduces the business load of the base station, and the resource allocation strategy of each agent can be updated in each round of iteration, which greatly improves the training convergence speed while achieving better performance.

It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

Example 4

According to an embodiment of the present application, a SWIPT-assisted QoS-based downlink resource allocation device is also provided. As shown in FIG. 5 , the device includes an acquisition module 542 , a selection module 544 and an allocation module 546 .

Based on the service type of the machine user device, the machine user device is divided into a tolerant machine user device and a critical machine user device; wherein the critical machine user device is a machine user device with a transmission reliability requirement higher than a preset requirement threshold, wherein the tolerant machine user device is a machine user device that needs to complete the transmission task of a periodically generated load.

The acquisition module 542 is configured to acquire a state observation value of a current environment state of a communication link between a tolerant machine user device and a machine device gateway.

The selection module 544 is configured to select a resource allocation strategy for the tolerant machine user device based on the state observation value of the current environment state and using a resource allocation model constructed based on a neural network.

The allocation module 546 allocates downlink resources to the tolerant machine user equipment based on the selected resource allocation strategy.

The downlink resource allocation device in this embodiment can implement the downlink resource allocation method in the above-mentioned embodiment, so it will not be described in detail here.

Example 5

An embodiment of the present application provides a SWIPT-assisted QoS-based downlink resource allocation system, as shown in Figure 6, including: a base station (BS) 52, a human user equipment (HUE) 60, a machine device gateway (MTCG) 54, and a machine user equipment, wherein the machine user equipment includes a critical machine user equipment (CMTCD) 56 and a tolerant machine user equipment (TMTCD) 58.

SWIPT is set on the machine user device. SWIPT is used to embed M2M communication, giving the machine user device the ability to obtain stable and reliable energy from the radio frequency environment and perform information decoding functions at the same time.

The spectrum subband K=H∪S indicates that each orthogonal spectrum subband is pre-allocated to a human user device or a critical machine user device, where H={1, 2, ..., H} represents, S={1, 2, ..., S} represents the critical machine user device; in addition, M={1, 2, ..., M} represents the number of machine device gateways or clusters; and the tolerant machine user device is represented by N={1, 2, ..., N}.

The base station 52 is used to establish an H2H communication link with the human user device 60, pre-allocate spectrum sub-bands and transmission power, and broadcast a priori information such as channel gain information and QoS indicators of all H2H links to the machine device gateway 54 within the coverage area.

The machine device gateway 54 is used to form a cluster with the machine user devices and establish multiple M2M communication links, and pre-allocate spectrum sub-bands, transmission power and power split ratios for critical M2M links. One machine device gateway and several machine user devices form a cluster.

The machine tool gateway (MTCG) 54 may be the downlink resource allocation device in the above embodiment, and may implement the downlink resource allocation method in the above embodiment, and therefore, it will not be described in detail here.

Example 6

An embodiment of the present application further provides a storage medium, which is configured to store program codes for executing the downlink resource allocation method in the above embodiment.

Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

Simulation test

为15dBm，发射功率和功率分流比分割等级L和Z分别为10和5，H2H业务SINR阈值

为7dB，关键型M2M业务SINR阈值

为5dB，中断概率

不超过0.01，容忍型M2M业务载荷传输时间约束T为100ms，载荷大小V为3×1024bytes，此外，考虑到有限精度数字信号处理的实际效果，规定每个用户的接收SINR值不超过30dB。对于执行算法所需参数，设置如下：折扣因子γ＝0.5，常数μ＝1/50，迭代次数3000次，且在前80％的迭代中贪婪因子∈由1线性衰减至0.01，每条容忍型M2M链路的神经网络均由3个全连接隐藏层组成，每层神经网络的神经元数目等于动作可选数目，即K×L×Z＝200个，Relu函数作为激活函数且RMSProp优化算法以0.001的学习率更新网络参数。The present application sets the following parameters: the base station coverage radius is 500m, the machine device gateway coverage radius is 30m, the number of H2H user devices is 2, the number of machine device gateways is 2, each machine device gateway has a critical M2M device within its coverage range, and the number of tolerant M2M devices within its coverage range increases from 0 to 5. The system bandwidth is 4MHz, the path loss is modeled as 128+37.6 log ₁₀ d, where d is in km, the shadow fading is modeled as a log-normal distribution with a standard deviation of 8dB, the fast fading is modeled as Rayleigh fading, the environmental noise power σ ² is -114dBm, the transmission power P _BS,h allocated by the base station to the H2H user device is 30dBm, the transmission power P _m,s allocated by the machine device gateway to the critical M2M user device is 23dBm, the power split ratio ρ _m,s is 0.8, and the maximum transmission power allocated to the tolerant M2M user device

The transmit power and power split ratio L and Z are 10 and 5 respectively, and the H2H service SINR threshold is

7dB, critical M2M service SINR threshold

is 5dB, the outage probability

The transmission time constraint T of the tolerant M2M service load is 100ms, and the load size V is 3×1024 bytes. In addition, considering the actual effect of finite precision digital signal processing, the receiving SINR value of each user is stipulated to be no more than 30dB. The parameters required for executing the algorithm are set as follows: discount factor γ = 0.5, constant μ = 1/50, number of iterations 3000 times, and the greed factor ∈ linearly decays from 1 to 0.01 in the first 80% of iterations. The neural network of each tolerant M2M link consists of 3 fully connected hidden layers, and the number of neurons in each layer of the neural network is equal to the number of optional actions, that is, K×L×Z=200. The Relu function is used as the activation function and the RMSProp optimization algorithm updates the network parameters at a learning rate of 0.001.

The spectrum subband, transmission power and power split ratio allocation method proposed in this application is named as multi-agent SWIPT-assisted adaptive spectrum-power-ratio allocation method (MA-SWIPT-ASPRA) according to its characteristics, and is compared with three efficient intelligent resource allocation methods: (1) single-agent SWIPT-assisted adaptive spectrum-power-ratio allocation method (SA-SWIPT-ASPRA), which is an algorithm that only changes the resource allocation decision of multiple M2M links simultaneously distributedly executed in the resource allocation method disclosed in this application to the resource allocation decision of a single M2M link asynchronously executed; (2) non-SWIPT-assisted adaptive spectrum-power-ratio allocation method (MA-Non-SWIPT-ASPRA), which is an algorithm that only removes the SWIPT function on the basis of the resource allocation method disclosed in this application; (3) Q-learning-based SWIPT-assisted spectrum-power-ratio allocation method (QL-SWIPT-ASPRA), which is the most classic intelligent resource allocation scheme based on reinforcement learning.

Figure 7 shows the change in the total energy efficiency of M2M devices as the number of M2M devices connected to the system increases. As can be seen from the figure, as the number of M2M devices increases, the total energy efficiency of M2M devices first increases and then decreases. The reasons are as follows:

When there are only critical M2M devices in the network, that is, when the number of M2M devices is 2, there is almost no difference in the total energy efficiency achieved by the SWIPT and non-SWIPT solutions. This is because there is no interference caused by spectrum sharing in the network at this time, and the critical M2M devices have a stronger channel gain due to their close distance to the local machine device gateway, thereby obtaining a very high SINR value. In addition, in the SWIPT solution, the critical M2M devices use most of the energy for information decoding and only a small part of the energy for collection. The large order of magnitude difference makes the energy efficiency performance almost the same.

When the number of M2M devices in the network increases from 2 to 6, the total energy efficiency achieved by each solution increases accordingly and reaches the maximum value when the number of M2M devices is 6. This is because spectrum reuse can improve spectrum efficiency and thus further improve network energy efficiency. When the number of M2M devices is 6, there are 4 tolerant M2M links that reuse the spectrum resources occupied by 2 H2H links and 2 critical M2M links. Each solution can allocate a pre-occupied spectrum subband to each tolerant M2M link without generating additional intra-cluster interference. Therefore, the spectrum efficiency of the network reaches the highest at this time, further achieving the highest energy efficiency value.

When the number of M2M devices in the network increases from 6 to 12, the energy efficiency achieved by each scheme decreases. This is because excessive spectrum reuse will generate both intra-cluster interference and inter-cluster interference, resulting in the widespread existence of interfering links and a serious increase in power consumption, leading to poor energy efficiency performance.

From the enabling conditions of each solution, the energy efficiency of the solution without SWIPT drops rapidly when the number of M2M devices exceeds 6, while the other SWIPT solutions can maintain a high energy efficiency until the number of M2M devices exceeds 8. This is because compared with the solution without SWIPT, the SWIPT solution can convert a large amount of interference power into energy collection while slightly reducing the spectrum efficiency. Therefore, based on the trade-off between spectrum efficiency and power consumption, the SWIPT solution can have better performance in a certain interference environment. This performance degradation phenomenon becomes gentle as the number of M2M devices continues to increase, because the continued increase in the number of M2M devices will seriously damage the spectrum efficiency, which will continue to reduce the space for further performance degradation.

From the performance of each solution, the resource allocation solution proposed in this application achieves the highest energy efficiency, and when the number of M2M devices increases from 6 to 8, the energy efficiency achieved by this application hardly decreases, indicating that this solution has a better balance between spectrum efficiency and energy consumption. This is because this application uses a multi-agent setting to make the training process distributed, which can fully consider the conditions of each agent at each moment and make the most suitable resource allocation strategy for each agent. The reason why the single-agent solution of centralized training performs slightly worse is that the resource allocation solution it makes is more universal and can be applied to each agent, ignoring the situation of each agent at each moment to a certain extent. The reason why the Q learning solution performs poorly is that the network environment is complex, the state and action set are huge, and the traditional reinforcement learning table lookup method is inefficient, which will ignore some excellent resource allocation solutions during the iteration process.

Figure 8 shows the change in the QoS requirement satisfaction rate of H2H users as the number of tolerant M2M devices connected to the system increases. As can be seen from the figure, as the number of tolerant M2M devices increases, the QoS requirement satisfaction rate of H2H users decreases. This is because increasing the number of tolerant M2M devices will increase the number of links that reuse the spectrum occupied by the H2H link, which will increase the interference power at the receiving end of the H2H link, thereby reducing the SINR value at the receiving end. In addition, the QoS requirement satisfaction rate of H2H users in the non-SWIPT solution reacts more dramatically to the increase in the number of tolerant M2M devices, and the performance drops rapidly when only a few M2M devices exist. There are two reasons for this phenomenon. First, in the non-SWIPT scheme, the tolerant M2M link is more willing to share the same spectrum resources with the H2H link, because the M2M receiving end of the tolerant link is mostly far away from the base station, so it is less interfered by the base station, so it can obtain a higher energy efficiency value. The critical M2M link within the control range of the machine device gateway can also achieve higher energy efficiency when no other users reuse the spectrum sub-band occupied by itself; second, in the SWIPT scheme, the tolerant M2M link is more willing to share the same spectrum resources with the critical M2M link, because SWIPT can convert the received interference into energy, thereby improving energy efficiency. Among all the schemes, the resource allocation method proposed in this application achieves the best H2H user QoS demand satisfaction rate, which proves that this method performs better in ensuring user QoS requirements.

FIG9 shows the change in the probability of critical M2M link interruption as the number of access-tolerant M2M devices in the system increases. As can be seen from FIG9, as the number of tolerant M2M devices increases, the probability of critical M2M link interruption increases accordingly. This is based on the fact that more spectrum access leads to a higher probability of interruption. In addition, the SWIPT-free scheme performs best. In addition to the spectrum reuse priority described in FIG2, the critical M2M devices in the SWIPT-free scheme use all the energy they receive for information decoding, so the SINR value obtained is higher, thereby reducing the probability of interruption. However, the SWIPT scheme sacrifices a certain amount of link transmission reliability in order to obtain higher energy efficiency. The resource allocation method proposed in this application performs best in the SWIPT scheme, confirming the high reliability of the proposed scheme.

Referring to Figure 4, the change in the success rate of tolerant M2M user load transmission as the number of tolerant M2M devices connected to the system increases is introduced. As can be seen from the figure, as the number of tolerant M2M devices increases, the success rate of load transmission decreases. This is because too much spectrum reuse brings more interference and power consumption, thereby reducing the capacity of each link for transmitting loads. In addition, the reason why the SWIPT scheme performs the worst is that, under the premise of not having the energy collection function, each tolerant M2M link can only maintain a high level of energy efficiency by selecting a lower transmission power, but at the same time reduces the capacity. Under the premise of having the energy collection function, the SWIPT scheme has each tolerant M2M link that is more inclined to receive a certain amount of interference to obtain higher energy efficiency, so it is willing to choose a higher level of transmission power, and the link capacity also increases accordingly. The resource allocation method proposed in this application has the best performance among all the schemes, which further confirms the effectiveness of the proposed method.

The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims (10)

1. A SWIPT-assisted downlink resource allocation method is characterized by comprising the following steps:

obtaining a state observation value of a current environment state of a communication link between tolerant machine equipment user equipment and a machine equipment gateway;

based on the state observation value of the current environment state, selecting a resource allocation strategy for the tolerant machine user equipment by using a resource allocation model constructed based on a neural network;

allocating downlink resources for the tolerant machine user equipment based on the selected resource allocation policy;

wherein the tolerant machine user device is a machine user device that needs to complete a transmission task of a periodically generated payload.

2. The method of claim 1, prior to obtaining a state observation that is tolerant to a current environmental state of a communication link between a machine device user equipment and a machine device gateway, the method further comprising:

dividing the machine user equipment into tolerant machine user equipment and key machine user equipment based on the service type of the machine user equipment;

wherein the key machine user equipment is a machine user equipment with transmission reliability requirement higher than a preset requirement threshold.

3. The method of claim 2, wherein the resource allocation model is constructed based on the following method:

constructing a state function, wherein the state function is a set of state observations;

constructing an action function, wherein the action function is a set of downlink spectrum resources, a transmission power level and a power split ratio;

constructing a reward function based on a balance of resource allocation optimization objectives and QoS constraints;

and constructing the resource allocation model based on the state function, the action function and the reward function.

4. The method of claim 3, wherein constructing the state function comprises: constructing the state function based on channel gain information of the communication link on each frequency spectrum sub-band, interference power size of the communication link on each frequency spectrum sub-band, load residue and transmission time residue of the communication link, current iteration number and greedy factor representing current environment exploration rate of the communication link.

5. The method of claim 3, wherein constructing a reward function comprises:

determining a reward function penalty provided by the tolerant machine user device based on the remaining amount of load and the remaining amount of transmission time of the tolerant machine user device;

determining a reward function penalty provided by the key machine user device based on an outage probability of the key machine user device;

determining a reward function penalty term of the human user equipment based on the signal-to-interference-and-noise ratio of the human user equipment;

constructing the reward function based on a total effective value of communication links of all machine user devices, a reward function penalty term provided by the tolerant machine user device, a reward function penalty term provided by the critical machine user device, and a reward function penalty term of the human user device.

6. The method of claim 4, wherein constructing the reward function based on a balance of resource allocation optimization objectives and QoS constraints comprises:

setting QoS constraint conditions;

taking the total energy value achieved by the tolerant machine user equipment and the critical machine user equipment as the resource allocation optimization target;

building the reward function based on the resource allocation optimization objective and the QoS constraints.

7. The method of claim 6, wherein setting the QoS constraints comprises:

setting the QoS constraint condition of the human user equipment to be that a signal to interference plus noise ratio (SINR) is larger than a set lowest threshold;

setting the QoS constraint condition of the tolerant machine user equipment to be that the transmission success rate of the load with the preset size V in the time constraint T is higher than a set success rate threshold;

setting the QoS constraint for a critical machine user equipment to be an outage probability not higher than a set outage threshold.

8. The method of claim 3, wherein after obtaining the state observation of the current environmental state of the communication link between the tolerant machine device user equipment and the machine device gateway, the method further comprises:

constructing an experience replay pool for storing training data for training the resource allocation model, wherein the training data comprises state observation values, reward values and selected resource allocation strategies at the current time and the next time;

the neural network comprises a training network and a target network, and the training network is trained in a random gradient descent mode by using data randomly extracted from the experience replay pool in each iteration; the target network is a fixed neural network, and the network parameters of the target network are updated to the training network parameters at the current moment at intervals.

9. The method of any one of claims 2 to 8, wherein the machine user equipment is provided with wireless communication energy carrying SWIPT for enabling the machine user equipment to obtain energy from a radio frequency environment and simultaneously decode information.

10. A SWIPT-assisted downlink resource allocation device is characterized by comprising:

an acquisition module configured to acquire a state observation of a current environmental state of a communication link between a tolerant machine user device and a machine device gateway;

a selection module configured to select a resource allocation policy for the tolerant machine user equipment based on the state observation of the current environmental state using a resource allocation model constructed based on a neural network;

the distribution module is used for distributing downlink resources to the tolerant machine user equipment based on the selected resource distribution strategy;

wherein the tolerant machine user device is a machine user device that needs to complete a transmission task of a periodically generated payload.

Images