CN115767562B - Service function chain deployment method based on reinforcement learning joint coordinated multi-point transmission - Google Patents

Abstract

The present invention discloses a service function chain deployment method based on reinforcement learning and joint cooperative multi-point transmission. First, the edge network model is described, the channel characteristics of the server and the user in the edge network are described, and beamforming is used to eliminate the communication interference between multiple servers and users; a mathematical model under the constraints of the number of server VNF instantiations, server processing capacity, physical link bandwidth, VNF routing and VNF migration budget is established; a long-term optimization problem is modeled; the long-term optimization problem is decoupled into a time slot optimization problem; finally, a sub-optimization problem for solving a reward function is established to reduce the complexity of action space search. The present invention uses a zero-forcing beamforming technology based on CoMP to eliminate wireless link interference between users, and then uses an Actor-Critic algorithm based on a natural gradient method to decouple the long-term dynamic SFC deployment problem into a time slot optimization problem, so as to solve it online.

Description

Service function chain deployment method based on reinforcement learning and joint cooperative multi-point transmission

Technical Field

The present invention belongs to the field of communication technology, and in particular relates to a service function chain deployment method based on reinforcement learning and joint collaborative multi-point transmission.

Background Art

Under the SDN (Software Defined Network, SDN) architecture, the 6G communication system is expected to improve the quality of network services through the emerging Network Function Virtualization (NFV) technology, so that service functions can be directly deployed in commercial servers using virtual machines or container technology without being deployed in dedicated hardware. This allows 6G communication system operators to flexibly expand the network scale directly on demand by increasing the number of commercial servers. Using NFV technology, the logical sequence of all service functions that a data packet has passed through can be cascaded to form an SFC, and through the orchestration of virtualized network functions (VNF), flexible customization and deployment of multiple services can be achieved.

At present, there are many studies on SFC deployment methods for traditional network architectures, such as content distribution networks and centralized cloud computing networks. Unlike traditional network architectures, deploying SFC in 6G edge networks can make users and VNFs located close to each other and obtain services nearby, thereby improving the service quality of computing-intensive and delay-sensitive services; at the same time, using NFV technology, edge servers can provide richer VNF orchestration combinations, thereby providing more complex and flexible service types.

At present, the research on SFC deployment methods in edge networks mainly focuses on static networks with single time slots, ignoring the interference characteristics of wireless channels in edge networks and the dynamic change characteristics of cache resources, computing resources and communication resources in edge servers. Therefore, the present invention proposes an SFC deployment method based on online Actor-Critic learning and joint CoMP beamforming in 6G wireless edge networks, and uses CoMP-based zero-forcing beamforming technology to eliminate wireless link interference between multiple SFCs.

Summary of the invention

The purpose of the present invention is to provide a service function chain deployment method based on reinforcement learning and joint cooperative multi-point transmission, which utilizes the CoMP-based zero-forcing beamforming technology to eliminate the wireless link interference between users, and then uses the Actor-Critic algorithm based on the natural gradient method to decouple the long-term dynamic SFC deployment problem into a time slot-by-time slot optimization problem, so as to solve it online.

The technical solution adopted by the present invention is a service function chain deployment method based on reinforcement learning combined with cooperative multi-point transmission, which is specifically implemented according to the following steps:

Step 1: Describe the edge network model, including the characteristics of edge servers, network virtual functions, users, and service function chains;

Step 2: Describe the channel characteristics between servers and users in the edge network, and use beamforming to eliminate communication interference between multiple servers and users;

Step 3: Establish a mathematical model under the constraints of the number of server VNF instantiations, server processing capacity, physical link bandwidth, VNF routing, and VNF migration budget;

Step 4: Model the long-term optimization problem based on the resource constraints established in steps 1-3;

Step 5: Construct a Markov decision process model MDP to decouple the long-term optimization problem into a time-slot optimization problem.

Step 6: Use the Actor-Critic reinforcement learning algorithm based on natural gradient to learn the optimal SFC deployment strategy online in each time slot;

Step 7: When searching the action space, establish a sub-optimization problem to solve the reward function, reduce the complexity of the action space search, and finally obtain the optimal solution.

The present invention is also characterized in that:

Step 1 is implemented as follows:

Step 1.1: In the edge network, an edge server is connected to a remote radio frequency module (RRH) and uses It represents both the nth edge server and the RRH of the server, where represents the set of servers in the edge network, and N represents the total number of servers in the edge network; the edge servers are connected to each other through X2 links, and each edge server can provide a variety of different virtual functions using virtual machine technology;

Step 1.2 Use represents the mth user in the edge network, represents the set of users in the edge network, M represents the total number of users, assuming that each user can only be served by one service function chain SFC, and SFC is defined as follows:

in, represents the first service function of the SFC of user m, represents the lth service function, Indicates the last service function and is designated as the baseband processing vBBU.

Step 2 is implemented as follows:

2.1、There is Rayleigh fading and path loss between user m and RRH. represents the channel matrix between user m and RRH numbered n, where represents an L _n ×L _m dimensional complex matrix, L _n represents the number of transmit antennas of RRH numbered n, and L _m represents the number of receive antennas of user m. Then the signal um _{,t received by user m in time slot t} can be expressed as:

in, represents the channel matrix between user m and all RRHs in the edge network in time slot t, where represents the channel matrix between user m and RRH numbered n, where (·) ^H represents the conjugate transpose of the matrix, Indicates the total number of antennas of all RRHs; It is represented as the beamforming matrix of all RRHs for user m, d _m is the number of data streams received by user m; I is used to represent the identity matrix, means that the mean is zero and the covariance is Gaussian random codebook; n _m,t is the covariance Gaussian white noise;

2.2. The received signal u _{m,t of} user m in step 2.1 can be continuously encoded based on the Gaussian random codebook, and the second term of the formula can be removed. Then the received data rate R _{m,t of user m in time slot t} can be expressed as:

2.3. Settings and P _m,n represent the service function processing power consumption and wireless transmission power consumption provided by edge server n to user m, respectively. Indicates whether user m uses the VNF instance vBBU of edge server n; then the beamforming matrix of all RRHs for user m Should meet:

in Tr(·) represents the trace operation of the matrix;

2.4. Use the zero-forcing beamforming technology of RRH to eliminate the wireless interference between SFCs. That is, stack the channel matrices of all users and then perform QR decomposition:

in, Each item in can be expressed as: is an orthogonal basis, and is a full-rank upper triangular matrix, and the remaining matrix blocks of the upper triangle are arbitrary non-zero matrices. Therefore, the beamforming matrix of user m can be expressed as in

2.5. To eliminate interference, the following conditions must be met: and In the formula The first L _m rows of S _m,t are related to the received data rate and can be simplified to in H _m,t can be simplified to H _m,t =Q _m,t R _m,m,t , and the effective beamforming matrix ∑ _m,t can be defined as: Let the matrix Then step 2.3 to establish constraints is equivalent to:

Constraint 1:

Constraint 2:

2.6. The received data rate R _m,t in step 2.2 needs to be greater than or equal to the data rate threshold R _m,th in order to perform correct data decoding, that is:

Constraint 3: R _m,t =log ₂ | _ILm +∑ _m,t |≥R _m,th .

Step 3 is implemented as follows:

3.1、Assume represents the set of edge servers that can provide service function f. Assume that each service function can only be deployed on one edge server, that is:

Constraint 4: in Indicates service function Whether it is deployed on edge server n, represents whether the service function f is provided in the edge server n in time slot t, and and Satisfies: Constraint 5:

3.2. The total data rate of the service flows processed by a VNF instance cannot exceed the processing capacity of the VNF instance. Right now:

Constraint 6:

3.3. The total data rate of service flows on a link cannot exceed its link bandwidth Right now:

Constraint 7: in, express and Whether they are deployed on edge servers n and s respectively;

3.4. In time slot t, only when and When both are 1, Only then can it be 1; then and The relationship between can be described as:

Constraint 8:

3.5 Definition is the service migration cost of edge servers n and s, then the total service migration cost of the system cannot exceed the migration threshold C _mig,th , that is:

Constraint 9:

Step 4 is implemented according to the following steps:

4.1、Define the total system overhead including data flow overhead and power consumption overhead;

4.2. First define Then the RRH wireless transmission power consumption is Then define P _f,n as the energy consumption of edge server n when enabling service function f, Maintain service functionality for edge servers n The total cost of deploying SFC in the system in time slot t is:

Among them, η is the trade-off coefficient between data flow overhead and power consumption overhead. In the above formula, the first term represents the data flow overhead between edge servers, the second term represents the power consumption overhead of enabling service functions, the third term represents the power consumption overhead of providing service functions for users, and the fourth term represents the wireless transmission power consumption of RRH for beamforming;

4.3. Step 4.2 establishes the cost of deploying SFC in a single time slot t. On this basis, the long-term dynamic SFC deployment cost is defined as the average cost of each time slot in the entire deployment process. T represents the total number of time slots in the deployment process, and It means to find the minimum value of the long-term dynamic SFC deployment cost, that is:

Among them, the variables in C _t R _m,t 、 P _f, and ∑ _m,t are subject to the constraints 1-9 established in steps 3 and 2. By solving Get the specific deployment results of SFC for each time slot

Step 5 is implemented according to the following steps:

5.1. Establishing MDP quadruple The state space It contains four elements, namely, the wireless channel matrix between the user and RRH, the processing capacity of the VNF instance, the link bandwidth between the edge servers, and the SFC deployment result of the previous time slot, namely:

5.2. Define actions

5.3 Definition is the reward function corresponding to (s _t ,a _t ); if the action a _t taken cannot find a feasible solution, the reward function is set to a small negative number;

5.4. Given an action a _t , find the maximum value of the reward function r(s _t , a _t ) and record the problem of finding the maximum reward function as Right now:

in, Express request The given parameters will be Convert to Used to solve the specific deployment results of SFC in each time slot

Step 6 is implemented according to the following steps:

6.1. Use an Actor neural network to output the deployment strategy, use a Critic neural network to evaluate each strategy through the Q-value approximation method, and use the neural network w to approximate the action value function, that is, _Qw ( _st , _at ) ^≈Qπ ( _st , _at ), where _Qw ( _st , _at ) represents the expected reward that can be obtained in subsequent states after taking action _at in state _st , and ^Qπ ( _st , _at ) is the action value function;

6.2. Experience replay and target network technology are used to improve training stability. At this time, the loss function of the Critic network can be defined as:

in, represents the expectation operator, is the experience replay pool, w′ is the model of the target network in time slot t, is an estimate of the expected value of the average return;

6.3. Find the gradient of Loss(w) with respect to w, and the update method of w is as follows:

Among them, α _c is the learning rate of the Critic network, and I is the number of samples obtained in the experience replay pool;

6.4. Based on the parameterized strategy π _θ , the expected value of the average return is defined as follows:

in, represents the steady-state distribution of state s;

6.5. Using the natural gradient method to train the Actor network, the update method of the network model θ becomes: Where α _a represents the learning rate of the Actor network, F(θ) is the Fisher information matrix, represents the gradient of J(π _θ ) with respect to θ;

6.6. Integrate the Actor network and the Critic network so that the neural network training is carried out along the natural gradient direction, making the neural network model approach the global optimum.

Step 7 is implemented according to the following steps:

7.1. Yes and To relax the Transformed into a convex problem; therefore, the L _p (0<p<1) norm penalty function is introduced to force the slack variable to be an integer between 0 and 1; let get The asymptotically optimal subproblem of as follows:

Among them, σ is the penalty parameter, δ is an arbitrarily small positive number,

variable and Satisfy constraints

7.2. The iterative method for defining the penalty parameter is as follows: δ _v+1 =ηδ _v (η>1), so that the penalty term P _δ (y) converges to 0 at a linear rate;

7.3. Due to The penalty term in is non-convex, resulting in It is difficult to solve, so we use the continuous convex approximation SCA technology to Transform it into a convex problem and solve it. The penalty term is expanded by the first order Taylor, that is Where y ^v is the optimal solution of the last SCA iteration, is the gradient of P _δ (y) with respect to y at point y ^v ;

7.4. In the v+1th SCA iteration, Finally it becomes a convex problem, namely:

7.5. According to the above steps, P _1-S is An asymptotically optimal solution of is obtained by solving P _1-S The optimal solution of the reward function is further obtained, and finally the deployment results of the SFC in each time slot are obtained according to the maximum reward.

The beneficial effect of the present invention is that the service function chain deployment method based on reinforcement learning and joint collaborative multi-point transmission can complete the long-term dynamic deployment of SFC in a non-interference manner under the premise of ensuring user service quality, and further reduce the operating overhead of the edge server and the wireless transmission power consumption overhead of RRH during the deployment process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a system model of SFC deployment with joint CoMP beamforming in a wireless edge network proposed by the present invention;

FIG2 is a schematic diagram of an algorithm flow of an SFC online deployment based on Actor-Critic learning in a wireless edge network proposed by the present invention.

DETAILED DESCRIPTION

The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

In conjunction with Figures 1 and 2, Figure 1 is a schematic diagram of a system model for SFC deployment with joint CoMP beamforming in a wireless edge network, which includes two SFC instances, an edge network, a mapping relationship between multiple service functions and edge servers, and two CoMP beams for two different users; Figure 2 is a flow chart of the Actor-Critic algorithm used to solve the problem of SFC online deployment in a wireless edge network. This embodiment describes in detail a method for online deployment of SFC based on the Actor-Critic algorithm in a wireless edge network.

The service function chain deployment method based on reinforcement learning combined with cooperative multi-point transmission of the present invention is specifically implemented according to the following steps:

Step 1: Describe the edge network model, including the characteristics of edge servers, network virtual functions, users, and service function chains;

Step 1 is implemented as follows:

Step 1.1: In the edge network, an edge server is connected to a remote radio head (RRH) and uses represents the set of servers in the edge network, and N represents the total number of servers in the edge network. The edge servers are connected to each other through X2 links, and each edge server can provide a variety of different virtual functions (for example, cache, computing, and firewall) using virtual machine technology.

Step 1.2 Use represents the mth user in the edge network, Represents the set of users in the edge network, and M represents the total number of users. Assume that each user can only be served by one service function chain (SFC), and define SFC as follows:

in, represents the first service function of the SFC of user m, represents the lth service function, The last service function is indicated and designated as baseband processing (Virtualized Building Base band Unit, vBBU).

Step 2: Based on step 1, describe the channel characteristics of the server and the user in the edge network, and use beamforming to eliminate communication interference between multiple servers and users;

Step 2 is implemented as follows:

2.1 There is Rayleigh fading and path loss between user m and RRH. represents the channel matrix between user m and RRH numbered n, where represents an L _n ×L _m dimensional complex matrix, L _n represents the number of transmit antennas of RRH numbered n, and L _m represents the number of receive antennas of user m. Then the signal um _{,t received by user m in time slot t} can be expressed as:

in, represents the channel matrix between user m and all RRHs in the edge network in time slot t, where represents the channel matrix between user m and RRH numbered n, where (·) ^H represents the conjugate transpose of the matrix, Indicates the total number of antennas of all RRHs; It is represented as the beamforming matrix of all RRHs for user m, d _m is the number of data streams received by user m; I is used to represent the identity matrix, means that the mean is zero and the covariance is Gaussian random codebook; n _m,t is the covariance Gaussian white noise;

2.2 The received signal u _{m,t of} user m in step 2.1 can be continuously encoded based on the Gaussian random codebook, and the second term of the formula can be removed. Then the received data rate R _m,t of user m in time slot t can be expressed as:

2.3 Settings and P _m,n represent the service function processing power consumption and wireless transmission power consumption provided by edge server n to user m, respectively. Indicates whether user m uses the VNF instance vBBU of edge server n; then the beamforming matrix of all RRHs for user m Should meet:

in Tr(·) represents the trace operation of the matrix.

2.4 Use the zero-forcing beamforming technology of RRH to eliminate the wireless interference between SFCs, that is, stack the channel matrices of all users and then perform QR decomposition:

in, Each item in can be expressed as: is an orthogonal basis, and is a full-rank upper triangular matrix, and the remaining matrix blocks in the upper triangle are non-zero matrices. Therefore, the beamforming matrix of user m can be expressed as in

2.5 To eliminate interference, the following conditions must be met: and In the formula The first L _m rows of S _m,t are related to the received data rate and can be simplified to in H _m,t can be simplified to H _m,t = Q _m,t R _m,m,t . The effective beamforming matrix ∑ _m,t can be defined as: Let the matrix Then step 2.3 to establish constraints is equivalent to:

Constraint 1:

Constraint 2:

2.6 The received data rate R _m,t in step 2.2 needs to be greater than or equal to the data rate threshold R _m,th for correct data decoding, that is:

Constraint 3: R _m,t ＝log ₂ |I _Lm +∑ _m,t |≥R _m,th

Step 3: Establish a mathematical model under the constraints of the number of server VNF instantiations, server processing capacity, physical link bandwidth, VNF routing, and VNF migration budget;

Step 3 is implemented as follows:

3.1 Design represents the set of edge servers that can provide service function f. Assume that each service function can only be deployed on one edge server, that is:

Constraint 4: in Indicates service function Whether it is deployed on edge server n, Indicates whether the service function f is provided in the edge server n in time slot t. And and Satisfies: Constraint 5:

3.2 The total data rate of service flows processed by a VNF instance cannot exceed the processing capacity of the VNF instance Right now:

Constraint 6:

3.3 The total data rate of service flows on a link cannot exceed its link bandwidth Right now:

Constraint 7: in, express and Whether to deploy them on edge servers n and s respectively.

3.4 In time slot t, only when and When both are 1, Only then can we take 1. Then and The relationship between can be described as:

Constraint 8:

3.5 Definition is the service migration cost of edge servers n and s, then the total service migration cost of the system cannot exceed the migration threshold C _mig,th , that is:

Constraint 9:

Step 4: Model the long-term optimization problem based on the resource constraints established in steps 1-3;

Step 4 is implemented according to the following steps:

4.1 Definition The total system overhead includes data flow overhead and power consumption overhead.

4.2 First define Then the RRH wireless transmission power consumption is Then define P _f,n as the energy consumption of edge server n when enabling service function f. Maintain service functionality for edge servers n The total cost of deploying SFC in the system in time slot t is:

Among them, η is the trade-off coefficient between data flow overhead and power consumption overhead. In the above formula, the first term represents the data flow overhead between edge servers, the second term represents the power consumption overhead of enabling service functions, the third term represents the power consumption overhead of providing service functions for users, and the fourth term represents the wireless transmission power consumption of RRH for beamforming.

4.3. Step 4.2 establishes the cost of deploying SFC in a single time slot t. On this basis, the long-term dynamic SFC deployment cost is defined as the average of the system cost of each time slot during the entire deployment process. T represents the total number of time slots during the deployment process, and It means to find the minimum value of the long-term dynamic SFC deployment cost, that is:

Among them, the variables in C _t R _m,t 、 P _f,n and ∑ _m,t are subject to the constraints 1-9 established in steps 3 and 2. By solving Get the specific deployment results of SFC for each time slot

Step 5: Construct a Markov Decision Processes model MDP to decouple the long-term optimization problem into a time-slot optimization problem.

Step 5 is implemented according to the following steps:

5.1 Establishing MDP Quadruple The state space It contains four elements, namely, the wireless channel matrix between the user and the RRH, the processing capacity of the VNF instance, the link bandwidth between the edge servers, and the SFC deployment result of the previous time slot, namely:

5.2 Defining Actions This is because the original action space contains variables The dimension is too high. Therefore, the action space is reduced in dimension.

5.3 Definitions is the reward function corresponding to (s _t ,a _t ); if the action a _t taken cannot find a feasible solution, the reward function is set to a small negative number.

5.4 Given an action a _t , find the maximum value of the reward function r(s _t , a _t ) and record the problem of finding the maximum reward function as Right now:

in, Express request The given parameters will be Convert to Used to solve the specific deployment results of SFC in each time slot

Step 6: Use the Actor-Critic reinforcement learning algorithm based on natural gradient to learn the optimal SFC deployment strategy online in each time slot;

Step 6 is implemented according to the following steps:

6.1 Use an Actor neural network to output the deployment strategy, and use a Critic neural network to evaluate each strategy through the Q-value approximation method. Use the neural network w to approximate the action value function, that is, Q _w (s _t , a _t ) ≈ Q ^π (s _t , a _t ), Q _w (s _t , a _t ) represents the expected return of each subsequent state after taking action a _t in state s _t , and Q ^π (s _t , a _t ) is the action value function.

6.2 In order to break the temporal correlation between samples, experience replay and target network technology are used to improve training stability. At this time, the loss function of the Critic network can be defined as:

in, represents the expectation operator, is the experience replay pool, w′ is the model of the target network in time slot t, is an estimate of the expected value of the average return.

6.3 Find the gradient of Loss(w) with respect to w, and the update method of w is as follows:

Among them, _αc is the learning rate of the Critic network, and I is the number of samples obtained in the experience replay pool.

6.4 Based on the parameterized strategy π _θ , the expected value of the average return is defined as follows:

in, represents the steady-state distribution of state s.

6.5 In order to avoid J(π _θ ) falling into the local optimum when training along the standard gradient direction, the natural gradient method is used to train the Actor network, and the update method of the network model θ becomes: Where α _a represents the learning rate of the Actor network, F(θ) is the Fisher information matrix, represents the gradient of J(π _θ ) with respect to θ.

6.6 The algorithm flow is shown in Figure 2. The Actor network and the Critic network are integrated so that the neural network training is carried out along the natural gradient direction, making the neural network model approach the global optimum.

Step 7: When searching the action space, establish a sub-optimization problem for solving the reward function to reduce the complexity of the action space search, and solve the reward function set in step 5.3 to obtain The asymptotically optimal solution of .

Step 7 is implemented according to the following steps:

7.1 pairs and To relax the is converted to a convex problem; however, the convex problem obtained after relaxation cannot guarantee that the optimal solution is a 0-1 integer solution, so the relaxed problem is not equivalent to the original problem. Therefore, the L _p (0<p<1) norm penalty function is introduced to force the relaxation variable to be a 0-1 integer. get The asymptotically optimal subproblem of as follows:

Among them, σ is the penalty parameter, δ is an arbitrarily small positive number, variable and Satisfy constraints

7.2 The iterative method for defining the penalty parameter is as follows: δ _v+1 =ηδ _v (η>1), so that the penalty term P _δ (y) converges to 0 at a linear rate.

7.3 Due to The penalty term in is non-convex, resulting in It is difficult to solve, so we use the Successive Convex Approximation (SCA) technique to solve Transform it into a convex problem and solve it. The penalty term is expanded by the first order Taylor, that is Where y ^v is the optimal solution of the last SCA iteration, It is the gradient of _Pδ (y) with respect to y at point ^yv .

7.4 In the v+1th SCA iteration, Finally it becomes a convex problem, namely:

7.5 According to the above steps, P _1-S is An asymptotically optimal solution of is obtained by solving P _1-S The optimal solution of the reward function is further obtained, and finally the deployment results of the SFC in each time slot are obtained according to the maximum reward.

Claims (5)

1. A method for deploying a service function chain based on reinforcement learning and coordinated multi-point transmission, characterized in that the method is implemented in the following steps: Step 1: Describe the edge network model, including the characteristics of edge servers, network virtual functions, users, and service function chains; Step 2: describe the channel characteristics of the server and the user in the edge network, and use beamforming to eliminate the communication interference between multiple servers and users; Step 2 is specifically implemented according to the following steps: 2.1、There is Rayleigh fading and path loss between user m and RRH. represents the channel matrix between user m and RRH numbered n, where represents an L _n ×L _m dimensional complex matrix, L _n represents the number of transmit antennas of RRH numbered n, and L _m represents the number of receive antennas of user m. Then the signal um _{,t received by user m in time slot t} can be expressed as: in, represents the channel matrix between user m and all RRHs in the edge network in time slot t, where represents the channel matrix between user m and RRH numbered n, where (·) ^H represents the conjugate transpose of the matrix, Indicates the total number of antennas of all RRHs; It is represented as the beamforming matrix of all RRHs for user m, d _m is the number of data streams received by user m; I is used to represent the identity matrix, means that the mean is zero and the covariance is Gaussian random codebook; n _m,t is the covariance Gaussian white noise; 2.2. The received signal u _{m,t of} user m in step 2.1 can be continuously encoded based on the Gaussian random codebook, and the second term of the formula can be removed. Then the received data rate R _{m,t of user m in time slot t} can be expressed as: 2.3. Settings and P _m,n represent the service function processing power consumption and wireless transmission power consumption provided by edge server n to user m, respectively. Indicates whether user m uses the VNF instance vBBU of edge server n; then the beamforming matrix of all RRHs for user m Should meet: in Tr(·) represents the trace operation of the matrix; 2.4. Use the zero-forcing beamforming technology of RRH to eliminate the wireless interference between SFCs. That is, stack the channel matrices of all users and then perform QR decomposition: in, Each item in can be expressed as: is an orthogonal basis, and is a full-rank upper triangular matrix, and the remaining matrix blocks of the upper triangle are arbitrary non-zero matrices. Therefore, the beamforming matrix of user m can be expressed as in 2.5. To eliminate interference, the following conditions must be met: and In the formula The first L _m rows of S _m,t are related to the received data rate and can be simplified to in H _m,t can be simplified to H _m,t =Q _m,t R _m,m,t , and the effective beamforming matrix ∑ _m,t can be defined as: Let the matrix Then step 2.3 to establish constraints is equivalent to: Constraint 1: Constraint 2: 2.6. The received data rate R _m,t in step 2.2 needs to be greater than or equal to the data rate threshold R _m,th in order to perform correct data decoding, that is: Constraint 3: R _m,t = log ₂ |I _Lm +∑ _m,t |≥R _m,th ; Step 3: Establish a mathematical model under the constraints of the number of server VNF instantiations, server processing capacity, physical link bandwidth, VNF routing, and VNF migration budget; The step 3 is specifically implemented according to the following steps: 3.1、Assume represents the set of edge servers that can provide service function f. Assume that each service function can only be deployed on one edge server, that is: Constraint 4: in Indicates whether the service function _fl ^m is deployed on the edge server n, Indicates whether the service function f is provided in the edge server n in time slot t, and and Satisfies: Constraint 5: 3.2. The total data rate of the service flows processed by a VNF instance cannot exceed the processing capacity of the VNF instance. Right now: Constraint 6: 3.3. The total data rate of service flows on a link cannot exceed its link bandwidth Right now: Constraint 7: in, represents _fl ^m and Whether they are deployed on edge servers n and s respectively; 3.4. In time slot t, only when and When both are 1, Only then can it be 1; then and The relationship between can be described as: Constraint 8: 3.5 Definition is the service migration cost of edge servers n and s, then the total service migration cost of the system cannot exceed the migration threshold C _mig,th , that is: Constraint 9: Step 4: Model the long-term optimization problem based on the resource constraints established in steps 1-3; The step 4 is specifically implemented according to the following steps: 4.1、Define the total system overhead including data flow overhead and power consumption overhead; 4.2. First define Then the RRH wireless transmission power consumption is Then define P _f,n as the energy consumption of edge server n when enabling service function f. The energy consumption of maintaining service function _fl ^m for edge server n is then the total cost of deploying SFC in the system in time slot t is: Among them, η is the trade-off coefficient between data flow overhead and power consumption overhead. In the above formula, the first term represents the data flow overhead between edge servers, the second term represents the power consumption overhead of enabling service functions, the third term represents the power consumption overhead of providing service functions for users, and the fourth term represents the wireless transmission power consumption of RRH for beamforming; 4.3. Step 4.2 establishes the cost of deploying SFC in a single time slot t. On this basis, the long-term dynamic SFC deployment cost is defined as the average of the system cost of each time slot in the entire deployment process. T represents the total number of time slots in the deployment process, and It means to find the minimum value of the long-term dynamic SFC deployment cost, that is: Among them, the variables in C _t P _f,n and ∑ _m,t are subject to the constraints 1-9 established in steps 3 and 2. By solving Get the specific deployment results of SFC for each time slot Step 5: Construct a Markov decision process model MDP to decouple the long-term optimization problem into a time-slot optimization problem. Step 6: Use the Actor-Critic reinforcement learning algorithm based on natural gradient to learn the optimal SFC deployment strategy online in each time slot; Step 7: When searching the action space, establish a sub-optimization problem to solve the reward function, reduce the complexity of the action space search, and finally obtain the optimal solution. 2. According to the service function chain deployment method based on reinforcement learning and coordinated multi-point transmission in claim 1, it is characterized in that the step 1 is specifically implemented according to the following steps: Step 1.1: In the edge network, an edge server is connected to a remote radio frequency module (RRH) and uses It represents both the nth edge server and the RRH of the server, where represents the set of servers in the edge network, and N represents the total number of servers in the edge network; the edge servers are connected to each other through X2 links, and each edge server can provide a variety of different virtual functions using virtual machine technology; Step 1.2 Use represents the mth user in the edge network, represents the set of users in the edge network, M represents the total number of users, assuming that each user can only be served by one service function chain SFC, and SFC is defined as follows: Where, f ₁ ^m represents the first service function of the SFC of user m, and _fl ^m represents the lth service function. Indicates the last service function and is designated as the baseband processing vBBU. 3. The service function chain deployment method based on reinforcement learning and coordinated multi-point transmission according to claim 1 is characterized in that step 5 is specifically implemented according to the following steps: 5.1. Establishing MDP quadruple The state space It contains four elements, namely, the wireless channel matrix between the user and the RRH, the processing capacity of the VNF instance, the link bandwidth between the edge servers, and the SFC deployment result of the previous time slot, namely: 5.2. Define actions 5.3 Definition is the reward function corresponding to (s _t ,a _t ); if the action a _t taken cannot find a feasible solution, the reward function is set to a small negative number; 5.4. Given an action a _t , find the maximum value of the reward function r(s _t , a _t ) and record the problem of finding the maximum reward function as Right now: in, Express request The given parameters will be Convert to Used to solve the specific deployment results of SFC in each time slot 4. The service function chain deployment method based on reinforcement learning and coordinated multi-point transmission according to claim 1 is characterized in that step 6 is specifically implemented according to the following steps: 6.1. Use an Actor neural network to output the deployment strategy, use a Critic neural network to evaluate each strategy through the Q-value approximation method, and use the neural network w to approximate the action value function, that is, _Qw ( _st , _at ) ^≈Qπ ( _st , _at ), where _Qw ( _st , _at ) represents the expected reward of each subsequent state after taking action _{at in state st .} Q ^π (s _t ,a _t ) is the action value function; 6.2. Experience replay and target network technology are used to improve training stability. At this time, the loss function of the Critic network can be defined as: in, represents the expectation operator, is the experience replay pool, w′ is the model of the target network in time slot t, is an estimate of the expected value of the average return; 6.3. Find the gradient of Loss(w) with respect to w, and the update method of w is as follows: Among them, α _c is the learning rate of the Critic network, and I is the number of samples obtained in the experience replay pool; 6.4. Based on the parameterized strategy π _θ , the expected value of the average return is defined as follows: in, represents the steady-state distribution of state s; 6.5. Using the natural gradient method to train the Actor network, the update method of the network model θ becomes: Where α _a represents the learning rate of the Actor network, F(θ) is the Fisher information matrix, represents the gradient of J(π _θ ) with respect to θ; 6.6. Integrate the Actor network and the Critic network so that the neural network training is carried out along the natural gradient direction, making the neural network model approach the global optimum. 5. The service function chain deployment method based on reinforcement learning and coordinated multi-point transmission according to claim 1 is characterized in that step 7 is specifically implemented according to the following steps: 7.1. Yes and To relax the Transformed into a convex problem; therefore, the L _p (0<p<1) norm penalty function is introduced to force the slack variable to be an integer between 0 and 1; let get The asymptotically optimal subproblem of as follows: Among them, σ is the penalty parameter, δ is an arbitrarily small positive number, variable and Satisfy constraints 7.2. The iterative method for defining the penalty parameter is as follows: δ _v+1 =ηδ _v (η>1), so that the penalty term P _δ (y) converges to 0 at a linear rate; 7.3. Due to The penalty term in is non-convex, resulting in It is difficult to solve, so we use the continuous convex approximation SCA technology to Transform it into a convex problem and solve it. The penalty term is expanded by the first order Taylor, that is Where y ^v is the optimal solution of the last SCA iteration, is the gradient of P _δ (y) with respect to y at point y ^v ; 7.4. In the v+1th SCA iteration, Finally it becomes a convex problem, namely: 7.5. According to the above steps, P _1-S is An asymptotically optimal solution of is obtained by solving P _1-S The optimal solution of the reward function is further obtained, and finally the deployment results of the SFC in each time slot are obtained according to the maximum reward.