CN114980327A - Network slice resource optimization method and device based on deep reinforcement learning - Google Patents

Abstract

The invention discloses a network slice resource optimization method and device based on deep reinforcement learning, wherein the method comprises the following steps: setting a network slice frame of a soft and hard mixing strategy; on the basis of ensuring the network slice SLA in the network slice window based on the network slice frame, maximizing the spectral efficiency and obtaining the network slice optimization problem; and based on the network slice optimization problem, under the condition of giving initial resource allocation, carrying out online network slice resource optimization and adjustment processing by using the DQN network to obtain a network slice resource optimization result. In the embodiment of the invention, the resource allocation can be carried out on the network slices with arbitrary QoS requirements based on a uniform optimization framework; the spectrum efficiency can be improved on the basis of guaranteeing different slice SLAs.

Description

Network slice resource optimization method and device based on deep reinforcement learning

Technical Field

The invention relates to the technical field of communication, in particular to a network slice resource optimization method and device based on deep reinforcement learning.

Background

In recent years, with the falling and research of the 5G technology, the vertical industries such as industrial internet of things, cellular internet of vehicles, enhancement/virtual reality and the like are applied and developed vigorously. Disparate applications bring differentiated performance requirements, such as the internet of vehicles requiring extremely high time delay and reliability, and augmented/virtual reality requiring high speed and low time delay. The different vertical application scenarios promote the 5G network to provide differentiated services on the same physical network, and put higher requirements on the scalability, availability and cost of the 5G network. Network slicing techniques are extensively studied in 3GPP and are considered to be subversive techniques that motivate 5G to provide flexible, customizable heterogeneous services on the same physical network.

The network slice is on the basis of a general physical network infrastructure, and different transmission tasks can be completed by a plurality of network slices through virtualization, isolation and sharing of wireless resources. At present, standard organizations, such as 3GPP, also widely discuss how to design a Radio Access Network (RAN) to support Network slicing, and develop a Service Level Agreement (SLA) guarantee research on the RAN side based on slice resource orchestration and management. The network slicing technology of the core network is mature, and can be realized through advanced computing, caching, virtualization container technology and the like. In contrast, RAN-side slicing is more challenging due to limited radio resources, heterogeneous service requirements, and dynamic wireless network environment. From the Radio Resource Management (RRM) point of view, RRM between multiple network slices is crucial to guarantee SLA agreement of network slices. The conventional MAC scheduler directly schedules each user at each Transmission Time Interval (TTI), and due to the existence of multiple slices, the conventional MAC scheduling problem actually becomes a multi-objective optimization problem, and is too complex to solve in actual deployment. Instead, multi-slice RRM under the SLA framework is decomposed into inter-slice and intra-slice RRM, which may be performed separately on different time scales. The intra-slice RRM is responsible for distributing intra-slice special resources to corresponding intra-slice users, focuses on TTI level data packet performance, and corresponds to a slice customized MAC scheduler. Inter-slice radio resource allocation, which is responsible for allocating the total radio resources to the individual slices at the beginning of each slice period window, focuses on the global guarantees of the slice SLA agreement. Therefore, how to optimize the inter-slice radio resource allocation to guarantee the global SLA of the network has become a new technical problem.

In the prior art, a resource allocation method and a resource allocation system for improving network quality in a multi-slice network are provided, which can make effective use of wireless resources. In order to achieve the aim, the invention adopts the following technical scheme to improve the resource allocation method of the network quality in the multi-slice network, which comprises the following contents that 1) the heterogeneous network to be tested is sliced, modeling is carried out aiming at the resource allocation problem influencing the network quality of each slice, and a resource allocation model of each slice is established; 2) processing the non-convex function in each resource allocation model to obtain an approximate convex function of each resource allocation model; 3) under the condition that tenants corresponding to the slices equally divide total subcarriers, solving an approximate convex objective function of each slice resource allocation model by adopting a continuous convex approximation algorithm according to the number of subcarriers of each slice, and determining an optimal slice resource allocation strategy and a scoring standard of each slice; 4) under the condition that the tenants corresponding to the slices do not equally divide the total subcarriers, solving the approximate convex objective function of the resource allocation model of each slice by adopting a non-cooperative game method according to the number of the subcarriers of each slice, and determining the optimal slice resource allocation strategy and scoring standard of each slice.

The second prior art provides a 5G network slice resource allocation method, which takes the requirement of user time delay as a main constraint condition to meet the service requirements of different users under different time delay requirements and improve the spectrum utilization rate.

The invention mainly comprises the following steps: 1) acquiring the communication delay sensitivity of a user: acquiring the sensitivity of a mobile user in a 5G wireless network slice to communication delay, and carrying out data quantization on the sensitivity; 2) user service request assignment: distributing service requests proposed by mobile users to different network slices according to service types; 3) allocating spectrum resources among slices: according to the generalized Kelly algorithm and the channel state of the current user, defining a cost function based on user transmission delay, iteratively solving network parameters related to base station spectrum resource allocation in the cost function under the objective of the lowest cost function, and allocating the base station spectrum resources to different network slices based on the parameters; 4) and (3) user spectrum resource allocation: further dividing the frequency spectrum resources of the base stations distributed to different network slices by utilizing a Lyapunov algorithm, so that each mobile user obtains the required frequency spectrum resources; 5) and (3) carrying out data transmission: and the user completes data transmission according to the acquired frequency spectrum resources.

The above prior art improves and improves the resource allocation of the slice network in a certain aspect. However, in a scenario where multiple heterogeneous service slices coexist, the following problems exist in the prior art: 1. due to the fact that multiple slices bring diversified quality of service (QoS) indexes, the traditional resource allocation problem actually becomes a multi-objective optimization problem, and is too complex to solve in actual deployment. The network module is designed based on QoS guarantee, so that the network module is highly coupled with QoS indexes, and the design principle of flattening and expanding a 5G network is violated; 2. the performance isolation between slices and the spectral efficiency are a difficult trade-off problem, and how to balance the relationship between the performance isolation between slices and the spectral efficiency is rarely considered in the prior art.

Disclosure of Invention

The invention aims to overcome the defects of the prior art, and provides a network slice resource optimization method and device based on deep reinforcement learning, which can be used for carrying out resource allocation on network slices with arbitrary QoS requirements based on a uniform optimization framework; the spectrum efficiency can be improved on the basis of guaranteeing different slice SLAs.

In order to solve the above technical problem, an embodiment of the present invention provides a method for optimizing network slice resources based on deep reinforcement learning, where the method includes:

setting a network slicing frame of a soft and hard mixing strategy;

on the basis of ensuring the network slice SLA in the network slice window based on the network slice frame, maximizing the spectral efficiency and obtaining the network slice optimization problem;

and performing online network slice resource optimization adjustment processing by using the DQN under the condition of giving initial resource allocation based on the network slice optimization problem to obtain a network slice resource optimization result.

Optionally, the network slice frame with the soft and hard hybrid policy includes:

dividing the network slice resource of the total bandwidth into a soft strategy and a hard strategy;

for a hard policy, allocating dedicated bandwidth resources to each network slice in a network slice window;

for the soft strategy, occupying public network slice resources in each network slice according to a preset scheduling priority and flow requirements;

wherein, the network slice resource of the total bandwidth is the sum of the dedicated bandwidth resource and the public network slice resource, and the formula is as follows:

wherein, w _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

w _c，k representing the resources allocated to the common network slice c in the network slice window k.

Optionally, the network slice frame for setting the software and hardware mixing policy further includes:

for the network slice isolation, the isolation is based on resource occupation; for a network slice m, defining the isolation of the network slice m in a network slice window k as follows:

wherein o is _m，k Representing the isolation of the network slice m in a network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k; w is a _c，m，k Representing a network slice m from a common network slice resource w _c，k The amount of resources occupied in; w is a _c，k Representing the resources allocated to the common network slice c in the network slice window k.

Optionally, on the basis of ensuring the network slice SLA in the network slice window based on the network slice frame, the expression for maximizing the spectral efficiency is as follows:

s.t.Q _m，k ＝f(d _m，k ，w _m，k ，w _c，m，k )；

wherein Q is _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k; s _k Represents the spectral efficiency in the network slice window k; d _m，k Representing the traffic demand of the network slice m in the network slice window k; f (.) represents SLA satisfaction rate Q _m，k Complex mathematical relations among flow demand, network slice resource allocation and network slice internal scheduling algorithms; alpha is alpha _m Representing network slicesA utility factor for m; β represents a utility factor corresponding to spectral efficiency;

the indication function is used for indicating whether the meeting rate of the network slice SLA meets the requirement or not;

representing the required isolation of the network slice m; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

w _c，k representing resources allocated to the public network slice c in the network slice window k; w represents the network slice resources of the total bandwidth; o. o _m，k Representing the isolation of the network slice m in a network slice window k; w is a _c，m，k Representing a network slice m from a common network slice resource w _c，k The amount of resources occupied in; r is _n，t Representing the rate of packets for user n at time t.

Optionally, the formula of the rate of the data packet at time t of the user n is as follows:

wherein, W _n，t Indicates the allocated bandwidth of user n at the tth TTI time; e represents the transmission error probability; q ^-1 (.) represents the inverse of a gaussian Q function; l _n，t Indicating the length of the codeword block in units of coincidence; gamma ray _n，t Indicating the reception of user n associated to base station b at time tReceiving the signal-to-interference-and-noise ratio of the signal; c _n，t Representing the channel dispersion for user n at time t.

Optionally, the formula of the signal to interference plus noise ratio of the received signal at time t of the user n associated with the base station b is as follows:

wherein, P _b Represents the transmit power of base station b; h _b，n，t Representing the channel gain between the user n and the base station b at the time t; n is a radical of ₀ A power spectral density representing gaussian white noise; w _b Network slice resources representing the total bandwidth of base station b.

Optionally, the SLA satisfaction rate includes:

for throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by the user within the slice to the minimum transmission speed requirement, which is as follows:

for the delay and reliability indexes, in a network slice window, the SLA satisfaction rate of the network slice m is defined as a proportional characteristic of successful transmission of a data packet within a specified maximum delay, which is specifically as follows:

wherein the content of the first and second substances,

a set of users representing an mth network slice;

represents the minimum data rate required for network slice m; r is _n，t Indicating packets for user n at time tA rate;

indicating the transmission reliability of user n within the kth network slice window under the maximum transmission delay constraint.

Optionally, the performing, by using the DQN network, an online network slice resource optimization adjustment process includes:

defining the DQN network as:

wherein, mu _m，k Representing the resource utilization rate of the network slice m in the network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

o _m，k representing the isolation of the network slice m in a network slice window k; q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k;

in the kth network slice window, the network slice m returns extra resources to the public network slice, and the network slice m +1 requests resources from the public network slice to increase the special resources of the network slice;

for a network slice, its action space is defined as:

wherein 0 < a ¹ ＜…＜a ⁱ I is an integer;

suppose network slice m is at the kthNetwork slice window moves as

Then in the next network slice window, the amount of dedicated resources is: w is a _m，k+1 ＝w _m，k +a _m，k ；

After the action definition of one network slice is given, the definition of the action space of the agent is the permutation and combination of all the network slice actions, and the mathematical expression is as follows:

after the system utility is given, the definition of the reward is given directly according to the system utility.

Optionally, the definition of directly giving the reward according to the system utility is specifically as follows:

wherein, [ x ]] ⁺ Max (0, x), ρ represents a penalty factor for not meeting the isolation requirement, ρ > 0; s _k /S _max Expressing normalized spectral efficiency, the exponential reward function is to satisfy the rate Q at SLA _m，k Training the network more efficiently near 1; q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k; alpha is alpha _m A utility factor representing a network slice m; β represents a utility factor corresponding to spectral efficiency; o _m，k Representing the isolation of the network slice m in a network slice window k;

a set of network slices is represented as,

S _k represents the spectral efficiency in the network slice window k; s _max Represents the maximum spectral efficiency;

and representing an indication function for representing whether the SLA satisfaction rate of the network slice meets the requirement or not.

In addition, an embodiment of the present invention further provides a device for optimizing network slice resources based on deep reinforcement learning, where the device includes:

setting a module: a network slice frame for setting a soft and hard mixing strategy;

an optimization problem module: the network slice frame is used for maximizing the spectral efficiency and obtaining a network slice optimization problem on the basis of guaranteeing the network slice SLA in the network slice window based on the network slice frame;

an optimization and adjustment module: and the method is used for performing online network slice resource optimization adjustment processing by using the DQN under the condition of giving initial resource allocation based on the network slice optimization problem to obtain a network slice resource optimization result.

In the embodiment of the invention, the resource allocation can be carried out on the network slices with arbitrary QoS requirements based on a uniform optimization framework; the spectrum efficiency can be improved on the basis of guaranteeing different slice SLAs.

Other advantages are correspondingly embodied in the specific embodiments in the specification.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a flowchart illustrating a deep reinforcement learning-based network slice resource optimization method according to an embodiment of the present invention;

FIG. 2 is a schematic structural component diagram of a deep reinforcement learning-based network slice resource optimization apparatus in an embodiment of the present invention;

FIG. 3 is a block diagram of a network slice for a soft-hard hybrid strategy in an embodiment of the invention;

fig. 4 is a diagram of a DQN network architecture in an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example one

Referring to fig. 1, fig. 1 is a flowchart illustrating a method for optimizing network slice resources based on deep reinforcement learning according to an embodiment of the present invention.

As shown in fig. 1, a method for optimizing network slice resources based on deep reinforcement learning, the method includes:

s11: setting a network slice frame of a soft and hard mixing strategy;

in the specific implementation process of the present invention, the network slice frame for setting the software and hardware mixing policy includes:

dividing the network slice resources of the total bandwidth into a soft strategy and a hard strategy;

for a hard policy, allocating dedicated bandwidth resources to each network slice in a network slice window;

for the soft strategy, occupying public network slice resources in each network slice according to a preset scheduling priority and flow requirements; wherein, the network slice resource of the total bandwidth is the sum of the dedicated bandwidth resource and the public network slice resource, and the formula is as follows:

wherein, w _m，k Representing the allocated dedicated bandwidth resources per network slice m in a network slice window k；

A set of network slices is represented that is,

w _c，k representing the resources allocated to the common network slice c in the network slice window k.

Further, the network slice frame for setting the soft and hard hybrid policy further includes:

for the network slice isolation, the isolation is based on resource occupation; for a network slice m, defining the isolation of the network slice m in a network slice window k as follows:

wherein o is _m，k Representing the isolation of the network slice m in a network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k; w is a _c，m，k Representing a network slice m from a common network slice resource w _c，k The amount of resources occupied in; w is a _c，k Representing the resources allocated to the common network slice c in the network slice window k.

Specifically, a resource slicing framework with soft slices and hard slices mixed is provided in a 5G slicing network, and slicing resource allocation is performed by utilizing a DQN network, so that the purpose of guaranteeing slicing SLA in the whole army is achieved, and the spectrum efficiency is maximized on the basis. We first receive below a slice framework of soft-hard hybrid strategies, a wireless communication model, and involved SLA performance indicators.

In a pure hard slicing strategy, each network slice can only occupy the resources allocated to it, which can ensure complete isolation between network slices but reduces spectrum efficiency. In contrast, a soft network slicing approach driven by a flexible resource sharing strategy can maximize spectral efficiency, but performance isolation between network slices requires complex algorithm design and is difficult to guarantee even in practical systems. Therefore, a novel slicing framework of a soft-hard hybrid strategy is provided, which can simultaneously utilize the advantages of the hard slicing strategy and the soft slicing strategy to maximize the spectrum efficiency on the basis of ensuring the slice SLA to be isolated from certain performance. Specifically, a hard decision part in the hybrid slice frame ensures a certain degree of inter-slice isolation; soft strategies, i.e. the introduction of the setting of common slices, are used to guarantee SLA and improve spectral efficiency. In general, the slice framework of soft-hard hybrid strategies can be understood from both common slice setup and periodic slice resource adjustment.

Public network slice setting: a network slice frame diagram of a soft-hard hybrid strategy as shown in fig. 3; wherein the resources are divided into two parts corresponding to hard policies and soft policies: the hard policy corresponds to the dedicated resources allocated to each network slice; the soft policies correspond to resources allocated to the common network slice. For hard network slices, the resources allocated to each network slice can only be occupied by users in that network slice. For soft network slices, i.e., public network slices, users of all network slices can occupy resources of the public network slices according to traffic demands and priorities.

In one network slice resource allocation window, the worst case is considered, i.e. the maximum amount of resources required for a network slice greatly exceeds its average required amount of resources. For example, when a user moves to the edge of a cell or a new user joins a cell, the RB resources required for the corresponding network slice are larger than those required on average for the entire network slice window. With more specific numerical examples to illustrate: assuming that a network slice window includes 1000 TTIs, in the first 900 TTIs, the number of RBs required for network slice 1 is 30, and in the last 100 TTIs, the number of RBs required for network slice 1 increases to 40 because of the movement of the user. In order to guarantee service performance for network slice 1 users within the entire network slice window, the hard slicing strategy is to perform resource reservation according to the worst case, and according to the above example, the hard network slicing strategy allocates 40R base stations to slice 1 before the network slice window starts. It is clear that a pure hard slicing strategy will reduce the spectral efficiency, since the first 900 TTIs, C network slice 1 only needs 30 RBs, resulting in a resource waste of 10 RBs, corresponding to a resource proportion of 25%. However, the soft-hard hybrid strategy scheme can satisfy slice 1SLA and maximize spectral efficiency with only little performance isolation by reasonable resource allocation. For example, for the example described above, the hybrid strategy allocates 30 RB's dedicated resources to slice 1 and 15 RB's resources to the common network slice. This resource configuration enables the hard-sliced portion of resources to achieve SLA guarantees, as well as performance isolation, in most TTIs of the network slice window. Furthermore, when the resources of the hard network slice cannot satisfy the SLA satisfaction rate of network slice 1, the slice may occupy RB resources of the common network slice according to the scheduling priority. In the right part of fig. 3, an optimal resource allocation schematic diagram of the soft-hard hybrid strategy scheme is given, and under a reasonable allocation condition, the soft-hard hybrid strategy scheme can ensure slice SLA, and under the constraint of ensuring certain performance isolation, the spectrum efficiency is maximized.

Periodic network slice resource adjustment: as shown in fig. 3, the radio resource allocation is periodically adjusted according to the network slice window to sense and adapt to the dynamic wireless environment. For example, in the initial stage of sensing, in order to ensure SLA of a network slice, more resources may be allocated to a common network slice, and at this time, the isolation of the corresponding network slice is lower. With the increase of the network perception to the wireless environment, the network slice resource allocation can converge to an accurate scheme meeting the isolation requirement, and the SLA is ensured while the spectrum efficiency is maximized, corresponding to the slice resource allocation under the condition of the rightmost convergence in fig. 3.

Consider a classical cellular chip network based on Orthogonal Frequency Division Multiple Access (OFDMA) technology. Suppose that the sector network comprises a total of B base stations, denoted by the set

The user set is represented as

Network slice set is represented as

Each user can belong to one or more slices according to different services, and the user set defining the belonging slice m is represented as

And each base station provides wireless flow service with corresponding quality for users in the corresponding slice according to the service type and SLA requirements of the slice. The radio resource is divided in TTI unit in time domain, and is represented as t ∈ {1, 2, … }; similarly, the division is performed in units of RBs in the frequency domain. Suppose that the bandwidth of base station b can be divided into W _b Considering the average power allocation scheme, the signal-to-interference-and-noise ratio of the received signal at time t of user n associated to base station b can be expressed as:

wherein, P _b Represents the transmit power of base station b; h _b，n，t Representing the channel gain between the user n and the base station b at the time t; n is a radical of ₀ A power spectral density representing gaussian white noise; w _b Network slice resources representing the total bandwidth of base station b.

For classical large data packet traffic, such as Enhanced Mobile Broadband service (eMBB) traffic, the wireless domain has studied realistic rates for such traffic for a long time, and the wireless rate available to user n from base station b can be estimated directly according to shannon's formula. For Type services such as large-scale Machine Type Communication (mtc) and Ultra-high-Reliable Low-Latency Communication (urlclc), packet services corresponding to these services usually have the characteristic of small packet size, and the traditional shannon formula cannot accurately estimate the corresponding achievable rate. Therefore, the channel coding system theory of the limited block length is utilized to obtain the achievable rate theory of the small-size data packet. We present the achievable rates for large and small size packets in a wireless channel scenario, as follows:

wherein, W _n，t Indicating the allocated bandwidth of the user n at the tth TTI moment; epsilon represents the transmission error probability; q ^-1 (.) represents the inverse of a gaussian Q function; l _n，t Indicating the length of the codeword block in units of a coincidence; gamma ray _n，t Represents the signal-to-interference-and-noise ratio of the received signal of the user n associated to the base station b at the time t; c _n，t Representing the channel dispersion for user n at time t.

Typically, classical performance indicators in a network slice SLA include throughput, latency, and transmission reliability. For throughput, it can be directly obtained by accumulating the amount of data successfully transmitted over a period of time. However, for the latency of the packet, the packet queuing and service model of the user needs to be explicit.

In the present invention, the arrival distribution of sliced traffic packets is determined by the type of sliced traffic and the network has no a priori knowledge of the traffic arrival and traffic distribution of the sliced users. Assuming that each user corresponds to a cache data queue in the base station, when the data packet of the user arrives, the data packet is cached in a corresponding buffer area and is served according to a First-Come-First-Serve (FCFS) queuing strategy. Thus, the packet delay consists of two parts, namely the queuing delay and the transmission delay. Where the former is influenced by the scheduling policy and the latter is determined by the instantaneous data rate. E.g. delay D of ith packet of user n _n，i Can be calculated by the following formula:

D _n，i ＝W _n，i +δ _n，i ；

wherein, W _n，i The queuing delay of the ith data packet representing the user n; delta _n，i Which indicates the transmission delay of the ith packet of user n.

For transmission reliability, from the network transmission perspective, if the delay of a data packet exceeds the preset maximum delay, the data packet is discarded. The transmission reliability is the ratio of successfully transmitted data packets, so the transmission reliability of user n in slice m can be expressed as:

wherein the content of the first and second substances,

representing the maximum transmission delay allowed by the network slice m for its users.

For throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by the user within the slice to the minimum transmission speed requirement, which is as follows:

for the delay and reliability index, in a network slice window, the SLA satisfaction rate of the network slice m is defined as the proportional characteristic of successful transmission of the data packet within the specified maximum delay, which is specifically as follows:

wherein the content of the first and second substances,

to representA user set of an mth network slice;

represents the minimum data rate required for network slice m; r is _n，t Representing the rate of data packets of the user n at the time t;

indicating the transmission reliability of user n within the kth network slice window under the maximum transmission delay constraint.

Setting a network slicing frame of a soft and hard mixed strategy: the network resource with the total bandwidth W is divided into two parts, namely soft and hard parts. For hard policies, each slice m is allocated a dedicated bandwidth resource w in a network slice window k _m，k (ii) a For soft policies, the amount of resources allocated to a common slice is w _c，k Each network slice can occupy the resources of the public network slice according to the preset scheduling priority and the flow demand. The sum of the amount of resources of the dedicated resources and the common resources is the total bandwidth, i.e. the total bandwidth

Wherein, w _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

w _c，k representing the resources allocated to the common network slice c in the network slice window k.

For network slice isolation, a resource occupancy-based isolation is defined: for slice m, its isolation in one slice window k is defined as follows:

wherein o is _m，k Representing the isolation of the network slice m in a network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k; w is a _c，m，k Representing a network slice m from a common network slice resource w _c，k The amount of resources occupied in; w is a _c，k Representing the resources allocated to the common network slice c in the network slice window k.

S12: on the basis of ensuring the network slice SLA in the network slice window based on the network slice frame, maximizing the spectral efficiency and obtaining the network slice optimization problem;

in the specific implementation process of the present invention, on the basis of ensuring the network slice SLA in the network slice window based on the network slice frame, the expression of maximizing the spectral efficiency is as follows:

s.t.Q _m，k ＝f(d _m，k ，w _m，k ，w _c，m，k )；

wherein Q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k; s _k Represents the spectral efficiency in the network slice window k; d is a radical of _m，k Representing the traffic demand of the network slice m in the network slice window k; f (.) denotes the SLA satisfaction rate Q _m，k Multiplexing with flow demand, network slice resource allocation and network slice internal scheduling algorithmA miscellaneous mathematical relationship; alpha is alpha _m A utility factor representing a network slice m; β represents a utility factor corresponding to spectral efficiency;

representing an indication function, which is used for representing whether the SLA satisfaction rate of the network slice meets the requirement or not;

representing the required isolation of the network slice m; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

w _c，k representing resources allocated to the public network slice c in the network slice window k; w represents the network slice resources of the total bandwidth; o _m，k Representing the isolation of the network slice m in a network slice window k; w is a _c，m，k Representing a network slice m from a common network slice resource w _c，k The amount of resources occupied in; r is a radical of hydrogen _n，t Representing the rate of packets for user n at time t.

Further, the formula of the packet rate of the user n at the time t is as follows:

wherein, W _n，t Indicating the allocated bandwidth of the user n at the tth TTI moment; e represents the transmission error probability; q ^-1 (.) represents the inverse of a gaussian Q function; l _n，t Indicating the length of the codeword block in units of coincidence; gamma ray _n，t Represents the signal-to-interference-and-noise ratio of the received signal at time t of user n associated to base station b; c _n，t Representing the channel dispersion for user n at time t.

Further, the formula of the signal to interference plus noise ratio of the received signal at time t of the user n associated to the base station b is expressed as follows:

wherein, P _b Represents the transmit power of base station b; h _b，n，t Representing the channel gain between the user n and the base station b at the time t; n is a radical of ₀ A power spectral density representing gaussian white noise; w _b Network slice resources representing the total bandwidth of base station b.

Further, the SLA satisfaction rate includes:

for throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by the user within the slice to the minimum transmission speed requirement, which is as follows:

for the delay and reliability indexes, in a network slice window, the SLA satisfaction rate of the network slice m is defined as a proportional characteristic of successful transmission of a data packet within a specified maximum delay, which is specifically as follows:

wherein the content of the first and second substances,

a set of users representing an mth network slice;

minimum required to atmosphere network slice mA data rate; r is _n，t Indicating the data packet rate of the user n at the time t;

indicating the transmission reliability of user n within the kth network slice window under the maximum transmission delay constraint.

S13: and performing online network slice resource optimization adjustment processing by using the DQN under the condition of giving initial resource allocation based on the network slice optimization problem to obtain a network slice resource optimization result.

In the specific implementation process of the present invention, the performing online network slice resource optimization and adjustment processing by using a DQN network includes:

defining the DQN network as:

wherein, mu _m，k Representing the resource utilization rate of the network slice m in the network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

o _m，k representing the isolation of the network slice m in a network slice window k; q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k; in the k-th network slice window, the network slice m returns extra resources to the public network slice, and the network slice m +1 requests resources from the public network slice to increase the special resources of the public network slice; for a network slice, its action space is defined as:

wherein 0 < a ¹ ＜…＜a ⁱ I is an integer; suppose network slice m moves in the k-th network slice window as

Then in the next network slice window, the amount of dedicated resources is: w is a _m，k+1 ＝w _m，k +a _m，k (ii) a After the action definition of one network slice is given, the definition of the action space of the agent is the permutation and combination of all the network slice actions, and the mathematical expression is as follows:

after the system utility is given, the definition of the reward is given directly according to the system utility.

Further, the definition of directly giving the reward according to the system utility is specifically as follows:

wherein, [ x ]] ⁺ Max (0, x), ρ represents a penalty factor for not meeting the isolation requirement, ρ > 0; s. the _k /S _max Expressing normalized spectral efficiency, the exponential reward function is to satisfy the rate Q at SLA _m，k Training the network more efficiently near 1; q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k; alpha is alpha _m A utility factor representing a network slice m; β represents a utility factor corresponding to spectral efficiency; o _m，k Representing the isolation of the network slice m in a network slice window k;

a set of network slices is represented that is,

S _k represents the spectral efficiency in the network slice window k; s. the _max Represents the maximum spectral efficiency;

and representing an indication function for representing whether the SLA satisfaction rate of the network slice meets the requirement or not.

According to the state, action and reward designed by deep reinforcement learning; the first two hidden layers of DQN are fully connected layers consisting of 64 neurons, using the ReLU activation function. This is followed by two RseNet blocks, each containing two fully connected hidden layers, plus a direct path from the input to the output of the ResNet block, as shown in the dashed box of fig. 4.

In the embodiment of the invention, different QoS indexes are unified to the SLA satisfaction rate, and the resource allocation can be carried out on network slices with arbitrary QoS requirements based on a unified optimization framework; the system has high utility performance, is close to the optimal algorithm performance, and can improve the spectrum efficiency on the basis of ensuring different slice SLAs; the SLA performance of the web slice can be globally guaranteed even during training of deep reinforcement learning.

Example two

Referring to fig. 2, fig. 2 is a schematic structural component diagram of a device for optimizing network slice resources based on deep reinforcement learning according to an embodiment of the present invention.

As shown in fig. 2, an apparatus for optimizing network slice resources based on deep reinforcement learning, the apparatus comprising:

the setting module 21: a network slice frame for setting a soft and hard mixing strategy;

the optimization problem module 22: the network slice frame is used for maximizing the spectral efficiency and obtaining a network slice optimization problem on the basis of guaranteeing the network slice SLA in the network slice window based on the network slice frame;

the optimization and adjustment module 23: and the method is used for performing online network slice resource optimization adjustment processing by using the DQN under the condition of giving initial resource allocation based on the network slice optimization problem to obtain a network slice resource optimization result.

In the specific implementation process of the present invention, the implementation manner of the apparatus part of the present invention may refer to the embodiment of the method part, and is not described herein again.

In the embodiment of the invention, different QoS indexes are unified to the SLA satisfaction rate, and the resource allocation can be carried out on network slices with arbitrary QoS requirements based on a unified optimization framework; the system has high utility performance, is close to the optimal algorithm performance, and can improve the spectrum efficiency on the basis of ensuring different slice SLAs; the SLA performance of the web slice can be globally guaranteed even during training of deep reinforcement learning.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.

In addition, the method and the apparatus for optimizing network slice resources based on deep reinforcement learning according to the embodiments of the present invention are described in detail above, and a specific embodiment should be used herein to explain the principle and the implementation manner of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (10)

1. A network slice resource optimization method based on deep reinforcement learning is characterized by comprising the following steps:

setting a network slice frame of a soft and hard mixing strategy;

on the basis of ensuring the network slice SLA in the network slice window based on the network slice frame, maximizing the spectral efficiency and obtaining the network slice optimization problem;

and performing online network slice resource optimization adjustment processing by using the DQN under the condition of giving initial resource allocation based on the network slice optimization problem to obtain a network slice resource optimization result.

2. The method for optimizing network slice resources according to claim 1, wherein the setting of the network slice framework of the soft-hard hybrid policy comprises:

dividing the network slice resource of the total bandwidth into a soft strategy and a hard strategy;

for a hard policy, allocating dedicated bandwidth resources to each network slice in a network slice window;

for the soft strategy, occupying public network slice resources in each network slice according to a preset scheduling priority and flow requirements;

wherein, the network slice resource of the total bandwidth is the sum of the dedicated bandwidth resource and the public network slice resource, and the formula is as follows:

wherein, w _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented as,

w _c，k representing the allocated common network slice resources for each network slice c in the network slice window k.

3. The method according to claim 1, wherein the network slice framework for setting the soft-hard hybrid policy further comprises:

for the network slice isolation, the isolation is based on resource occupation; for a network slice m, defining the isolation of the network slice m in a network slice window k as follows:

wherein o is _m，k Representing the isolation of the network slice m in a network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k; w is a _c，m，k Representing a network slice m from a common network slice resource w _c，k The amount of resources occupied in; w is a _c，k Representing the resources allocated to the common network slice c in the network slice window k.

4. The method for optimizing network slice resources according to claim 1, wherein the expression for maximizing spectral efficiency on the basis of ensuring the network slice SLA in the network slice window based on the network slice framework is as follows:

s.t.Q _m，k ＝f(d _m，k ，w _m，k ，w _c，m，k )；

wherein Q is _m，k Is represented in a network slice windowThe SLA satisfaction rate for each network slice m in k; s _k Represents the spectral efficiency in the network slice window k; d _m，k Representing the traffic demand of the network slice m in the network slice window k; f (.) represents SLA satisfaction rate Q _m，k A complex mathematical relation between the flow demand and the resource allocation of the network slices and the scheduling algorithm in the network slices; alpha is alpha _m A utility factor representing a network slice m; β represents a utility factor corresponding to spectral efficiency;

representing an indication function, which is used for representing whether the SLA satisfaction rate of the network slice meets the requirement or not;

representing the required isolation of the network slice m; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

w _c，k representing resources allocated to the public network slice c in the network slice window k; w represents the network slice resources of the total bandwidth; o _m，k Representing the isolation of the network slice m in a network slice window k; w is a _c，m，k Representing network slices m from a common network slice resource w _c，k The amount of resources occupied in; r is a radical of hydrogen _n，t Representing the rate of packets for user n at time t.

5. The method of claim 4, wherein the rate of packets for user n at time t is formulated as follows:

wherein, W _n，t Indicates the allocated bandwidth of user n at the tth TTI time; e represents the transmission error probability; q ^-1 (.) represents the inverse of a gaussian Q function; l _n，t Indicating the length of the codeword block in units of coincidence; gamma ray _n，t Represents the signal-to-interference-and-noise ratio of the received signal at time t of user n associated to base station b; c _n，t Representing the channel dispersion for user n at time t.

6. The method of claim 5, wherein the formula of the SINR of the received signal at time t of user n associated to base station b is as follows:

wherein, P _b Represents the transmit power of base station b; h _b，n，t Representing the channel gain between the user n and the base station b at the time t; n is a radical of ₀ A power spectral density representative of gaussian white noise; w _b Network slice resources representing the total bandwidth of base station b.

7. The method of claim 4, wherein the SLA satisfaction rate comprises:

for throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by the user within the slice to the minimum transmission speed requirement, which is as follows:

for the delay and reliability index, in a network slice window, the SLA satisfaction rate of the network slice m is defined as the proportional characteristic of successful transmission of the data packet within the specified maximum delay, which is specifically as follows:

wherein the content of the first and second substances,

a set of users representing an mth network slice;

represents the minimum data rate required for network slice m; r is a radical of hydrogen _n，t Representing the rate of data packets of the user n at the time t;

indicating the transmission reliability of user n within the kth network slice window under the maximum transmission delay constraint.

8. The method of claim 1, wherein the performing online optimization and adjustment of network slice resources by using the DQN network comprises:

defining the DQN network as:

wherein, mu _m，k Representing the resource utilization rate of the network slice m in the network slice window k; w is a _m，k Representing the allocated dedicated bandwidth resources for each network slice m in the network slice window k;

a set of network slices is represented that is,

o _m，k representing the isolation of the network slice m in a network slice window k; q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k;

in the k-th network slice window, the network slice m returns extra resources to the public network slice, and the network slice m +1 requests resources from the public network slice to increase the special resources of the public network slice;

for a network slice, its action space is defined as:

wherein 0 < a ¹ ＜…＜a ⁱ I is an integer;

suppose network slice m moves in the k-th network slice window as

Then in the next network slice window, the amount of dedicated resources is: w is a _m，k+1 ＝w _m，k +a _m，k ；

After the action definition of one network slice is given, the definition of the action space of the agent is the permutation and combination of all network slice actions, and the mathematical expression is as follows:

after the system utility is given, the definition of the reward is given directly according to the system utility.

9. The method for optimizing network slice resources according to claim 8, wherein the definition of directly providing rewards according to system utility is specifically as follows:

wherein, [ x ]] ⁺ Max (0, x), ρ represents a penalty factor for not meeting the isolation requirement, ρ > 0; s _k /S _max Expressing normalized spectral efficiency, the exponential reward function is to satisfy the rate Q at SLA _m，k Training the network more efficiently near 1; q _m，k Representing the SLA satisfaction rate for each network slice m in the network slice window k; alpha is alpha _m A utility factor representing a network slice m; β represents a utility factor corresponding to spectral efficiency; o _m，k Representing the isolation of the network slice m in a network slice window k;

a set of network slices is represented that is,

S _k represents the spectral efficiency in the network slice window k; s _max Represents the maximum spectral efficiency;

and representing an indication function for representing whether the SLA satisfaction rate of the network slice meets the requirement or not.

10. An apparatus for optimizing network slice resources based on deep reinforcement learning, the apparatus comprising:

setting a module: a network slice frame for setting a soft and hard mixing strategy;

an optimization problem module: the network slice frame is used for maximizing the spectral efficiency and obtaining a network slice optimization problem on the basis of guaranteeing the network slice SLA in the network slice window based on the network slice frame;

an optimization and adjustment module: and the method is used for performing online network slice resource optimization adjustment processing by using the DQN under the condition of giving initial resource allocation based on the network slice optimization problem to obtain a network slice resource optimization result.

Images