CN110381541B - Smart grid slice distribution method and device based on reinforcement learning - Google Patents

Abstract

The invention discloses a smart grid slice distribution method based on reinforcement learning, which is characterized by comprising the following steps of: classifying the power business of the intelligent power grid according to the business type; corresponding the classifications to different slices; and constructing a reinforcement learning model of the intelligent power grid slice according to the service index of the intelligent power grid, and completing the distribution of the intelligent power grid slice through the reinforcement learning model to realize the resource scheduling management of the intelligent power grid. Classifying the service types of the intelligent power grid, corresponding the classifications to different slices, and completing the distribution of the intelligent power grid slices through the constructed reinforcement learning model of the intelligent power grid slices. Therefore, the integration problem of the 5G network slicing technology and the intelligent power grid based on reinforcement learning is solved.

Description

A smart grid slice allocation method and device based on reinforcement learning

Technical field

This application relates to the field of network resource allocation for power wireless communications, specifically to a smart grid slice allocation method based on reinforcement learning, and also to a smart grid slice allocation device based on reinforcement learning.

Background technique

With the advent of the 5G era of high-speed ubiquity, low power consumption, and low latency, communications in human society have gradually become smoother. Network slicing is considered one of the important key technologies of 5G networks. It divides a single physical network into multiple independent logical networks to support various vertical multi-service networks and distribute them in different business scenarios according to their characteristics. to adapt to different service needs. Using network slicing technology can greatly save deployment costs and reduce network occupancy.

Driven by the growth in energy and power demand, the world's power grid has entered the smart grid era from traditional networks. Combined with the new round of energy revolution, development in the communications field and the global Internet strategic concept, 5G network slicing technology has the possibility of being applied to smart grid services for the first time. The technical characteristics of 5G network slicing have the characteristics of customizable slices, safe and reliable isolation between slices, and unified slice management for carrying wireless business applications for the power grid. It also has the advantages of fast networking, high efficiency and economy, and has broad applications in power systems. prospect. Therefore, the integration of 5G network slicing technology based on reinforcement learning and smart grid is an urgent problem that needs to be solved.

Contents of the invention

This application provides a smart grid slicing allocation method based on reinforcement learning to solve the integration problem of 5G network slicing technology and smart grid based on reinforcement learning.

This application provides a smart grid slice allocation method based on reinforcement learning, which is characterized by including:

Classify the power business of smart grid according to business type;

Correspond to different slices according to the classification;

A reinforcement learning model of smart grid slices is constructed according to the service indicators of the smart grid. Through the reinforcement learning model, the allocation of smart grid slices is completed and the resource scheduling management of the smart grid is realized.

Preferably, the power services of the smart grid are classified according to business types, including:

The power business of smart grid is divided into control category, information collection category and mobile application category according to the business type.

Preferably, the classification corresponds to different slices, including:

The control class corresponds to the uRLLC slice, the information collection class corresponds to the mMTC slice, and the mobile application class corresponds to the eMBB slice.

Preferably, the reinforcement learning model of the smart grid is constructed. Specifically, the Q _-learning algorithm is used to construct the reinforcement learning model of the smart grid.

Preferably, constructing a reinforcement learning model for smart grid slicing includes: constructing reinforcement learning models for the wireless access side and the core network side respectively.

Preferably, the reinforcement learning model for constructing smart grid slices includes:

Define the state space as S={s ₁ , s ₂ ,..., s _n };

Action space A is defined as A={a ₁ ,a ₂ ,..., _an };

The reward function is R={s,a}, and P(s,s ^* ) represents the transition probability from state s to s';

At any time, the slice controller in state s can choose action a, receive an immediate reward R _t , and will also move to the next state s'. The process of the Q-learning algorithm can be expressed by the following updated formula,

where α is the learning rate, and is the discount accumulation of all instant rewards R _t ,

By updating the Q value within a long enough duration and adjusting the values of α and γ, we can ensure that Q(s,a) can eventually converge to the value of the optimal strategy, that is,

This application also provides a smart grid slice distribution device based on reinforcement learning, which is characterized by including;

Classification unit, which classifies the power business of smart grid according to business type;

A classification and slicing corresponding unit, which corresponds the classification to different slices;

The model building unit constructs a reinforcement learning model of smart grid slices according to the service indicators of the smart grid; through the reinforcement learning model, the allocation of smart grid slices is completed and the resource scheduling management of the smart grid is realized.

This application provides a method of allocating smart grid slices based on reinforcement learning. By classifying the business types of smart grids, the classifications correspond to different slices, and the allocation of smart grid slices is completed through the constructed reinforcement learning model of smart grid slices. . This solves the integration problem of 5G network slicing technology based on reinforcement learning and smart grid.

Description of drawings

Figure 1 is a schematic flow chart of a smart grid slice allocation method based on reinforcement learning provided by an embodiment of the present application;

Figure 2 is a schematic diagram of the slicing architecture in the smart grid scenario involved in the embodiment of this application;

Figure 3 is a schematic diagram of the relationship between slicing and the three types of services of the smart grid involved in the embodiment of the present application;

Figure 4 is the QoS indicator of a typical smart grid business slice involved in the embodiment of this application;

Figure 5 is a mapping of the smart grid slice resource management mechanism to RL involved in the embodiment of this application;

Figure 6 is a schematic diagram of a smart grid slice allocation device based on reinforcement learning provided by an embodiment of the present application.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described here. Those skilled in the art can make similar extensions without violating the connotation of the present application. Therefore, the present application is not limited by the specific implementation disclosed below.

Please refer to Figure 1. Figure 1 is a smart grid slice allocation method based on reinforcement learning provided by an embodiment of this application. The method provided by this application will be described in detail below in conjunction with Figure 1.

Step S101: Classify the power services of the smart grid according to service types.

First, the slicing architecture in the smart grid scenario based on this application is introduced, as shown in Figure 2.

Network slicing uses SDN technology to help decouple the control/data plane of the network and define open interfaces between the two to achieve flexible definition of network functions in network slicing. To meet the needs of this type of business, network slicing only includes network functions that support specific services. The power business can be divided into three categories: control (such as distribution automation, precise load control, etc.), information collection (such as power consumption information collection, transmission line monitoring, etc.), and mobile application (such as intelligent inspection, mobile operations, etc.) kind.

Step S102: Correspond the categories to different slices.

Figure 3 shows the relationship between the three major types of slices and the three types of services of the smart grid. The control class corresponds to the uRLLC slice, the information collection class corresponds to the mMTC slice, and the mobile application class corresponds to the eMBB slice.

Step S103: Construct a reinforcement learning model of smart grid slices according to the service indicators of the smart grid. Through the reinforcement learning model, the allocation of smart grid slices is completed and the resource scheduling management of the smart grid is realized.

Figure 4 shows the QoS (service) indicators of typical business slices of smart grids. This application considers the business plane, orchestration control plane and data plane. The business plane divides the business into elastic application (Elastic application) and real-time application (Real-Time application). Resilient applications can tolerate relatively large delays and have no minimum bandwidth requirements. Specific examples include car-mounted distributed power supply, video surveillance, user metering, etc. Real-time applications require a minimum level of performance guarantees from their networks. The main representative type is URLLC slicing service. Typical examples are power distribution automation, emergency communications, etc. The data plane stores data generated by the interaction between power equipment and the physical layer.

In this application, the orchestration control plane is mainly considered, and the access network SDN (Software Defined Network) controller and the core network SDN controller are introduced, respectively responsible for the network function (NF) management and coordination (such as services) of the access network and core network. Migration and deployment), they are equivalent to two different agents that can communicate with each other and complete coordination work together. Faced with various prior knowledge of service types, channel conditions, and user requirements on the service plane, the slice orchestration controller of the orchestration control plane completes the division of the slice network and divides it into radio access network (RAN) side slices and core networks. (CN) Side slice. The network slices on the RAN side and the CN side are managed by their respective SDN controllers, which are responsible for executing the algorithms on the respective network sides, which is the smart grid slice allocation method based on reinforcement learning proposed in this application.

The following describes the reinforcement learning models on the RAN side and CN side proposed in this application.

(1) RAN side radio resource slicing

Given a series of existing slices χ ₁ , χ ₂ ,..., χ _n , the vector χ is used to represent the set of existing slices as χ = {χ ₁ , χ ₂ ,..., χ _n }. These slices Shared aggregate bandwidth B; there is a series of service flows, represented by vector D = {d ₁ , d ₂ ,..., d _m }. Variable D is actually a collection of smart grid business flows. Faced with the multi-service characteristics of smart grids, each slice service needs to meet different QoS requirements. However, what kind of business this business flow is in the smart grid is unknown in advance, and the real-time demand changes of the business in the smart grid scenario are unstable. It can be seen that d _i (i∈M={1,2,..,m}) obeys a specific traffic model.

First, it is necessary to define the system state space, action space and reward function of the RAN side network. The interaction between the slice controller and the wireless environment is represented by the tuple [S,A,P(s,s ^* ),R(s,a)], where S represents the possible state set, A represents the possible action set, and P( s,s ^* ) represents the transition probability from state s to s', and R(s,a) is the reward associated with the action trigger in state s, which is fed back to the slice controller. The following is the mapping of wireless access side slice resource management to RL.

A. State space:

The state space is defined as a set of tuples S={s ^slice }. s ^slice is a vector that is used to represent the status of all currently available bearer-related power service slices, where the nth element is

B. Action space:

Facing the time-varying and unknown business traffic model, the reinforcement learning agent must allocate appropriate slice resources to the corresponding power business. The agent can decide how to perform actions at the next moment based on the current slice state and the reward function. Action space A is defined as A = {a ^bandwidth }, where a _bandwidth indicates that the agent allocates appropriate bandwidth to each logically independent slice to carry the corresponding service.

Since network slicing shares network resources between virtual networks, virtual network slices must be isolated from each other so that if the resources on one slice are insufficient to carry the current business and congestion or failure occurs, other slices will not be affected. Therefore, in order to ensure the isolation of slices and maximize the effectiveness of resource allocation, each slice is limited to carry only one kind of service at most:

Limit binary variables at the same time

C. Reward function

After the agent allocates a specific slice to a certain smart grid business, it will receive a comprehensive income, which we use as a reward for the system. Control power services have very strict requirements on communication delay and bit error rate. Communication failures or errors may affect the control execution of the power grid and lead to grid operation failures. For some mobile application services (such as inspection transmission video, high-definition video playback, etc.), a certain transmission rate is required, and there are higher requirements for communication bandwidth. Power supply reliability means continuous, sufficient, and high-quality power supply. For example, when the power supply reliability rate reaches 99.999% ("five nines"), it means that the average annual power outage time for electricity customers in the region will not exceed 5 minutes, and when this number reaches 99.9999% ("six nines") ”), the average annual power outage time for electricity customers in the region will be reduced to about 30 seconds. Due to limited spectrum resources on the RAN side, the optimal strategy should be selected when allocating slices to maximize the user's QoS needs.

Mainly considering the downlink situation, spectrum efficiency (SE) and delay (Delay) are used as evaluation indicators. The spectral efficiency of the system can be defined as:

According to the Shannon formula R=blog ₂ (1+(g ^BS→UE P)/σ ² ), the actual rate from the base station (BS) to the user can be obtained, where g ^BS→UE is the channel state (CSI) between the base station and the device ), obeys Rayleigh fading.

When describing users' QoS requirements, we introduce a utility function, which is a curve mapping between the bandwidth allocated to slice services and the performance perceived by users. In this article, we assume that the services carried by slices can be divided into elastic applications and real-time applications.

(a) Flexible application

For this type of application, there is no minimum bandwidth requirement because it can tolerate relatively large delays. The elastic flow utility model uses the following function:

where k is a tunable parameter that determines the shape of the utility function and ensures that when the maximum requested bandwidth is received, But even with very high bandwidth, the user satisfaction rating of this application is hardly 1. Therefore, we believe that the bandwidth allocated to this application type should not exceed the maximum bandwidth b _max even in the case of excessive network bandwidth.

(b)Real-time application

This application type of traffic requires a minimum level of performance guarantees from its network. If the allocated bandwidth drops below a certain threshold, QoS becomes unacceptable. Real-time applications are modeled using the following utility function:

Among them, k ₁ and k ₂ are adjustable parameters, which determine the shape of the utility function.

Define the rewards for the learning agent as follows:

R＝λ·SE+μ·U _e +ξ·U _rt

where λ, μ, ξ are the weights of SE, U _e and U _rt .

Therefore, mathematically speaking, our problem can be formulated as:

d _i (i∈M＝{1,2,..,m}) obeys a specific traffic model (*)

The key difficulty in solving problem (*) is that due to the existence of the traffic model, changes in business demand are unstable without being known in advance, that is, the real-time changes in business demand in the smart grid scenario are unknown.

(2) Core network slicing based on priority scheduling

Similarly, if we virtualize computing resources into VNFs per slice, then the problem of allocating computing resources to each VNF can be solved like wireless resource slicing. Therefore, in this part, we discuss another important issue, which is priority-based core network slicing of common VNFs. The mapping we use is slightly different from radio resource slicing to reflect the flexibility of RL. Similarly, the interaction between the slicing controller and the core network side is also represented by the quadruple [S, A, P (s, s ^* ), R (s, a)]. The following defines the appropriate RL elements for this slicing problem. mapping.

A. State space

There are related service function chains (SFCs) on the core network side, which have the same basic functions, but require different computing processing units (CPUs) and produce different results (such as service queuing time). For example, based on business value or other smart grid business-related characteristics, business flows can be divided into three categories (such as Category A, Category B, and Category C). The priority from Category A to Category C gradually decreases. Priority-based scheduling The rule is defined as follows: SFC I gives priority to Class A business flows, and SFC II treats Class A and Class B business flows equally, but serves Class C business flows with the lowest priority. SFC III treats all business streams equally. When scheduling based on priority, the queuing time of the service is generated.

The state space can be defined as T={T _q }, where T _q is a vector representing the queuing status of each element in the business set D. When N CPUs are used to calculate service d _i , the i-th element is T _qi , which represents the queuing time of service d _i , where i∈M={1,2,..,m}.

B. Action space

The final CPU usage of each SFC depends on the number of business flows it processes. With a limited number of CPUs, each type of business flow needs to be scheduled to the appropriate SFC, resulting in acceptable queuing time. Therefore, when processing service _di , it is necessary to select an appropriate number of CPUs N _CPU on the core network side. Therefore, the action space is defined as A _CPU ={a ^CPU }, where a ^CPU represents the incoming business _di (i∈M={1,2,..,m}), and the required CPU is selected when performing calculations. quantity.

C. Reward function

When defining the reward function, we first need a utility function U to represent the sensitivity of the current business to delay, and then define a new measurement "network request value" function W to represent the priority of the business.

As mentioned above, when describing elastic applications and real-time applications, we use utility functions:

To respectively characterize the QoS requirements of business _di . Compared with the RAN side, the difference is that the independent variable becomes the number n of CPUs required on the core network side when calculating the service _di . But this can only reflect the QoS requirements of different services. Due to the limited nature of computing resources, after allocating computing resources, reasonable scheduling rules are needed to reflect which business should be processed first. Therefore, the "network request value" function W is introduced to characterize the priority of the business. For any application service d _i , the network request value that needs to be satisfied is defined as:

W _i = 2 ^(p) U _i

Where p is the priority level of business d _i , and U _i is any element in the set of elastic applications and real-time applications, that is, U _i ∈ {U _e , U _kt }. The weight 2 ^(p) of a business request indicates the importance of the request relative to other requests. Define the reward function as:

R＝ _Wi

The above formula can only get the current priority of a certain business di _i . We need to get the priority queuing situation of a series of businesses, so we need to accumulate and maximize the long-term reward, that is

Figure 5 shows the mapping of smart grid slice resource management mechanisms to RL:

Next, the slice allocation method based on reinforcement learning in the context of the above model proposed in this application is introduced.

A reinforcement learning algorithm on the RAN and CN sides based on Q _-learning . Since the expressions of the RAN and CN side state sets, action sets and reward functions are slightly different above, and in this article, based on the mapping model from RL to RAN and CN we proposed, the Q _-learning algorithm has universal applicability, as For convenience, in this section we unify the state space as S={s ₁ ,s ₂ ,...,s _n }, the action space as A={a ₁ ,a ₂ ,..., _an }, and the reward The function is R={s,a}, and P(s,s ^* ) represents the transition probability from state s to s'.

The ultimate goal of the slicing controller is to find the optimal slicing policy π ^* , which is a mapping from the state set to the action set and needs to maximize the expected long-term discounted reward of each state:

The long-term discounted reward for state s is the discounted sum of rewards earned on the state trajectory and is given by:

R(s,π(s))+γR(s ₁ ,π(s ₁ ))+γ ² R(s ₂ ,π(s ₂ ))+...

Among them, γ is the discount factor (0<γ<1), which determines the current value corresponding to future rewards. The optimization objective in formula (*) represents the state value function of any strategy, which can be expressed as follows:

According to Bellman's optimality criterion, there exists at least one optimal strategy in a single environmental setting. Therefore, the state value function of the optimal policy is given by:

The state transition probability depends on many factors, such as traffic load, service arrival and departure rate, decision-making algorithm, etc., therefore, it may not be easily obtained either on the wireless side or the core network side. Therefore, model-free reinforcement learning is very suitable for deriving optimal policies because it does not require reward expectations and the state transition probabilities can be known as prior knowledge. Among various existing RL algorithms, we choose Q _-learning .

Taking the RAN side as an example, the slicing controller interacts with the wireless environment in a short discrete time period. The action-value function (also known as Q-value) of the state-action tuple (s, π(s)) can be expressed as Q(s, π(s)). Q(s,π(s)) is defined as the expected long-term discounted reward of state s when using policy π. Our goal is to find an optimization strategy that maximizes the Q value of each state s:

According to the Q _-learning algorithm, the slice controller can iteratively learn the optimal Q value based on existing information. At any time, a slice controller in state s can choose action a. This will be rewarded with an immediate reward R _t and will also move to the next state s'. The process of Q _-learning algorithm can be expressed by the following updated formula:

where α is the learning rate, and is the accumulation of discounts over all instant rewards R _t :

By updating the Q value within a long enough duration and adjusting the values of α and γ, we can ensure that Q(s,a) can eventually converge to the value of the optimal strategy, that is,

The entire slicing strategy is given by the following algorithm. Initially, the Q value is set to 0. Before the Q _-learning algorithm is applied, the slice controller performs initial slice allocation to different slices based on the power service traffic demand estimate of each slice. This is done for the state initialization of different slices. Existing wireless resource slicing solutions use bandwidth-based or resource-based provisioning to allocate wireless resources to different slices.

Since Q _-learning is an online iterative learning algorithm, it performs two different types of operations. In exploration mode, the slice controller randomly selects a possible action to enhance its future decisions. In contrast, in development mode, the slice controller prefers operations that it has tried in the past and found to work. We assume that the slice controller in state s explores with probability ε and exploits previously stored Q values with probability 1-ε. Not all actions are possible in any state. To maintain isolation between slices, the slice controller must ensure that the same physical resource block (PRB) is not allocated to two different slices (RAN side) .

Corresponding to the method provided by this application, this application also provides a smart grid slice allocation device 600 based on reinforcement learning, which is characterized by including;

Classification unit 610 classifies the power services of the smart grid according to service types;

The classification and slice corresponding unit 620 corresponds the classification to different slices;

The model building unit 630 constructs a reinforcement learning model of smart grid slices according to the service indicators of the smart grid; through the reinforcement learning model, allocation of smart grid slices is completed, and resource scheduling management of the smart grid is realized.

This application provides a method of allocating smart grid slices based on reinforcement learning. By classifying the business types of smart grids, the classifications correspond to different slices, and the allocation of smart grid slices is completed through the constructed reinforcement learning model of smart grid slices. . This solves the integration problem of 5G network slicing technology based on reinforcement learning and smart grid.

The above embodiments are only used to illustrate the technical solutions of the present invention but not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art can still make modifications or equivalent substitutions to the specific embodiments of the present invention. , and any modifications or equivalent substitutions that do not depart from the spirit and scope of the invention are within the scope of the claims of the pending invention.

Claims (6)

1. A smart grid slice allocation method based on reinforcement learning, which is characterized by including: Classify the power business of smart grid according to business type; Correspond to different slices according to the classification; A reinforcement learning model for smart grid slices is constructed according to the service indicators of the smart grid. Through the reinforcement learning model, the allocation of smart grid slices is completed and the resource scheduling management of the smart grid is realized. The reinforcement learning model of the smart grid slice includes: Reinforcement learning models on the wireless access side and core network side; The reinforcement learning model on the wireless access RAN side includes: given a series of existing slices χ ₁ , χ ₂ ,..., χ _n , the vector χ is used to represent the set of existing slices as χ = {χ ₁ , χ ₂ ,...,χ _n }, these slices share the aggregate bandwidth B; there is a series of business flows, represented by the vector D={d ₁ , d ₂ ,..., d _m }; the vector D is actually the intelligence A collection of power grid services; facing the multi-service characteristics of smart grid, each slice service needs to meet different QoS requirements; take the business d _i in the vector D, where i∈M={1,2,..,m }Subject to a specific traffic model; First, it is necessary to define the system state space, action space and reward function of the RAN side network; the interaction between the slice controller and the wireless environment is represented by the four-tuple means that/> Represents a state set,/> Represents an action set, Represents the transition probability from state s to s ^* , /> is the reward associated with the action trigger in state s,/> is fed back to the slice controller; The reinforcement learning model on the CN side of the core network includes: The interaction between the slice controller and the core network side is composed of four-tuple Represent; define the state space as/> T _q is a vector, representing the queuing status of each element in vector D; when N CPUs are used to process business d _i , the i-th element is T _qi , which represents the queuing time of business d _i , where i∈M={ 1,2,..,m}; When processing business d _i , an appropriate number of CPUs N _CPU needs to be selected on the core network side; therefore, the action space is defined as where a ^CPU represents the incoming business d _i , where i∈M={1,2,..,m}, and the number of CPUs required is selected when performing calculations; When defining the reward function, use the utility function: to respectively represent the QoS requirements of business d _i , where U _e (x) represents the elastic application utility model, U _rt (x) represents the real-time application utility model, and k, k ₁ and k ₂ are adjustable parameters. 2. The method according to claim 1, characterized in that the power services of the smart grid are classified according to service types, including: The power business of smart grid is divided into control category, information collection category and mobile application category according to the business type. 3. The method according to claim 1, characterized in that the classification corresponds to different slices, including: The control class corresponds to the uRLLC slice, the information collection class corresponds to the mMTC slice, and the mobile application class corresponds to the eMBB slice. 4. The method according to claim 1, characterized in that, the reinforcement learning model of the smart grid is constructed, specifically, a Q _-learning algorithm is used to construct the reinforcement learning model of the smart grid. 5. The method according to claim 4, further comprising: constructing a reinforcement learning model of the smart grid based on the Q _-learning algorithm, and allocating the smart grid slices, including: the state set is The action set is reward function is the reward associated with the triggering of action a corresponding to state s,/> Represents the transition probability from state s to s ^* ; At any time, the slice controller in state s can choose action a and get an immediate reward. At the same time, it will also transfer to the next state s'. The process of Q _-learning algorithm can be expressed by the following updated formula, Among them, γ represents the discount factor; t represents the time experienced from state s to s';a' represents the action under state s';A(s') represents the action space set under state s', α is the learning rate, and are all instant rewards/> accumulation of discounts, Among them, T represents the elapsed time, from time t=0 to the T-th time; by updating the Q value within the duration and adjusting the values of α and γ, it is ensured that Q(s,a) can eventually be in the optimal state. The strategy converges, and the value obtained by convergence is 6. A smart grid slice distribution device based on reinforcement learning, characterized by including: Classification unit, which classifies the power business of smart grid according to business type; A classification and slicing corresponding unit, which corresponds the classification to different slices; The model building unit constructs a reinforcement learning model of smart grid slices according to the service indicators of the smart grid; through the reinforcement learning model, the allocation of smart grid slices is completed, and the resource scheduling management of the smart grid is realized; the reinforcement learning of the smart grid slices Models, including: reinforcement learning models on the wireless access side and core network side; The reinforcement learning model on the wireless access RAN side includes: given a series of existing slices χ ₁ , χ ₂ ,..., χ _n , the vector χ is used to represent the set of existing slices as χ = {χ ₁ , χ ₂ ,...,χ _n }, these slices share the aggregate bandwidth B; there is a series of business flows, represented by the vector D={d ₁ , d ₂ ,..., d _m }; the vector D is actually the intelligence A collection of power grid services; facing the multi-service characteristics of smart grid, each slice service needs to meet different QoS requirements; take the business d _i in the vector D, where i∈M={1,2,..,m }Subject to a specific traffic model; First, it is necessary to define the system state space, action space and reward function of the RAN side network; the interaction between the slice controller and the wireless environment is represented by the four-tuple means that/> Represents a state set,/> Represents an action set, Represents the transition probability from state s to s ^* , /> is the reward associated with the action trigger in state s,/> is fed back to the slice controller; The reinforcement learning model on the CN side of the core network includes: The interaction between the slice controller and the core network side is composed of four-tuple Represent; define the state space as/> T _q is a vector, representing the queuing status of each element in vector D; when N CPUs are used to process business d _i , the i-th element is T _qi , which represents the queuing time of business d _i , where i∈M={ 1,2,..,m}; When processing business d _i , an appropriate number of CPUs N _CPU needs to be selected on the core network side; therefore, the action space is defined as where a ^CPU represents the incoming business d _i , where i∈M={1,2,..,m}, and the number of CPUs required is selected when performing calculations; When defining the reward function, use the utility function: to respectively represent the QoS requirements of business d _i , where U _e (x) represents the elastic application utility model, U _rt (x) represents the real-time application utility model, and k, k ₁ and k ₂ are adjustable parameters.