DE102021116590A1 - CONTINUOUS NETWORK SLICING IN A 5G MOBILE NETWORK OVER A DELAYED DEEP DETERMINISTIC POLICY GRADIENT - Google Patents

Abstract

Delayed deep deterministic policy gradient (DDPG) based continuous network slicing defines s at least two network slices in a central processing unit (CU) of a 5G network, and in different time steps identifies a state of each slice, determined from a reinforced learning policy of a critic model and an actor policy that assigns a scaling operation to the slices that allocates different computational resources to the appropriate virtual network functions (VNFs) in the CU for the slice based on state. The scaling process is applied to the CU and a cost result is monitored. The result is compared to a predetermined optimal result. Thereafter, gradients are calculated for each of the actor and critic models, considering a maximized reward based on a difference between the results. Notably, the actor politics gradient is applied less frequently over time than the critic model gradient.

Description

BACKGROUND OF THE INVENTION

Field of invention

The present invention relates to the field of cellular data communication and in particular to network slicing in a cellular telecommunication network of the fifth generation (5G).

Description of the related art

Cellular data communication is understood to be the exchange of data traffic over a cellular telecommunication network. Digital cellular data communication requires the existence of an underlying physical data communication infrastructure that is layered on top of a cellular network, as first introduced by second-generation digital cellular communication and, more recently, by the much more robust and reliable LTE (Long Term Evolution) cellular network of the fourth generation ( 4G) is occupied. Data communication network. With 4G LTE, the network architecture supports the connectivity of user equipment (UE) to different base stations (eNBs) that are clustered in different radio access networks (RANs), with the RANs each being coupled to the core network (CN).

The eNBs send and receive radio transmissions to each UE using the analog and digital signal processing functions of the LTE air interface via various multiple input multiple output (MIMO) antenna arrays. Each eNB also controls the low level operation of each coupled UE by sending the UE signal messages such as handover commands. Finally, eBN connects to the CN, also known as "Evolved Packet Core" (EPC), via an S1 protocol stack interface. It is noteworthy that each eBN can also be communicatively coupled to another, nearby eBN through an X2 interface in order to support signaling and packet forwarding during the handover of a communication with UE from eBN to eBN (cell to cell). The EPC, in turn, is a framework for the provision of convergent voice and data in the 4G LTE network. While 2G and 3G network architectures of the third generation process and switch voice and data via two separate subdomains - circuit switched (CS) for voice and packet switched (PS) for data - EPC combines voice and data on an Internet Protocol (IP) service architecture and Speech is treated as just another Internet Protocol (IP) application.

While 4G represented a huge leap in performance over 2D and 3G networks, 5G represents a huge improvement over 4G. 5G is able to achieve a data exchange capacity of almost thirteen terabytes - almost a twenty-fold improvement over 4G LTE. The CN of the 5G architecture reflects a significant change from the EPC of 4G. In the CN of 5G, the changes were abstractly reduced to the so-called "four modernizations". The first is “Information Technology” or “IT”, the second is the “Internet”, the third is “Extremely Simplified” and the fourth is “Service Based”. The most typical change in the network architecture of the CN is the service-based network architecture of the CN in order to separate the control plane from the user plane. Other technologies support network slicing and edge computing.

As for IT modernization, the characteristic of 5G architecture is the term Network Function Virtualization (NFV). NFV decouples software from hardware by replacing various network functions such as firewalls, load balancers and routers with virtualized instances that run as software. This eliminates the need to invest in many expensive hardware elements and can also accelerate installation times, thereby providing the customer with revenue-generating services more quickly. NFV enables the 5G infrastructure through virtualized appliances within the 5G network. This also includes network slicing technology, which allows multiple virtual networks to run at the same time. NFV can address other 5G challenges through virtualized compute, storage and network resources that are customized based on the applications and customer segments.

Some have referred to network slicing as a “key part” of 5G, which enables the full potential of the 5G architecture to be realized. Network slicing adds an extra dimension to the NFV domain by allowing multiple logical networks to run concurrently on a shared physical network infrastructure. Therefore, network slicing is becoming an integral part of the 5G architecture by creating end-to-end virtual networks that are both network and Include memory functions. Operators of a 5G network can then effectively manage different 5G use cases with different throughput, latency and availability requirements by partitioning network resources among several users or "tenants". With strategically coordinated network slicing and the optimized assignment of VNF instances (Virtual Network Function), the costs for operating a 5G architecture network can be optimized.

In terms of optimizing the configuration of various network slices, z-Ero-Touch and fully automated operations and management have become quintessential to take advantage of the potential gain of dynamic resource allocation in an NFV-enabled network slice. For this purpose, many have suggested the autonomous management and orchestration of VNFs, in which the CU “learns” to reconfigure resources, provide new VNF instances or outsource jobs to a central cloud. One notable proposal is for a Deep Reinforced Learning (DRL) based solution called a Parameterized Action Twin (PAT) Deep Deterministic Policy Gradient (DDPG) that uses the actor-critic method to learn network resources for VNFs online deployed, taking into account the current network status and the requirements of the deployed VNFs.

Notably, the PAT-DDPG solution outperforms all benchmark DRL schemes as well as heuristic greedy allocation in a variety of network scenarios. While capable of producing excellent results, a DDPG has its drawbacks. Like many reinforced learning algorithms, the training of DDPG can be unstable and heavily reliant on finding the right hyperparameters for the current task. This is caused by the fact that the algorithm continuously overestimates the Q-values of the critic (value) network. These estimation errors accumulate over time and can lead to the agent falling into a local optimum or catastrophic forgetting.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the present invention address deficiencies in the prior art relating to network slicing in a 5G network and provide a novel and unobvious method, system, and computer program product for continuous network slicing using delayed DDPG. In one embodiment of the invention, at least two network slices are defined, a cellular communication network being implemented in a CU of a 5G network. A network slicing function then identifies a state of each of the network slices in different time steps and determines a scaling process for a simultaneous time step in the allocation of different computing resources to corresponding VNFs in the CU from a reinforced learning policy that is assigned to one of the network slices for one of the network slices based on the identified status of the network slice. The network slicing function applies the determined scaling process further to the CU by assigning the various computing resources to the corresponding VNFs in the CU to the network slice.

It is important that the reinforced learning policy includes an actor policy and a critic model. The actor guideline takes into account the status of one of the network slices as input and delivers a specific scaling process as output, which assigns different computing resources for one of the network slices to the corresponding virtual network functions (VNFs) in the CU, based on the identified status of one of the Network slices. The critic model in turn takes into account the state of one of the network slices in combination with the determined scaling operation as input and delivers a statistical Q value as output. Optionally, the critic model can be embodied by an amalgamation of twin critic models, the statistical Q value being a minimization of the individual Q values generated by each of the twins. In particular, both the actor policy and the critic model can be implemented according to a deep neural network that learns itself based on the feedback applied to the network.

As soon as the scaling process has been applied, the network slicing function monitors a resource cost result of the determined scaling process in the CU and compares the monitored result with a predetermined optimal result for the determined scaling process. The network slicing function then determines a statistical Q-value in the critic model based on a difference between the monitored result and the optimal result and determines a gradient for each of the actor policies and the critic model that takes the determined statistical Q-value into account. Finally, the network slicing function applies the calculated gradient to each of the actor guidelines and the critic network model for use in a next particular scaling operation in a subsequent one of the time steps. However, the network slicing function applies the one calculated for the actor guideline Gradients at a rate that is less frequent than an application of the calculated gradient for the critic model.

In one aspect of the embodiment, the determined scaling operation is calculated taking into account a state space for that of the network slices, the state space including a number of new UE connections to that of the network slices, computing resources that each of the VNFs in the CU for the corresponding one of the network -Slices are assigned, determined. a delay status related to the latency cost for each of the network slices, a power status related to the energy cost for the use of the computing resources by each of the network slices, a number of users served in each of the network slices, and a Number of VNF instantiations in each of the network slices. In another aspect of the embodiment, the scaling process is part of a vertical scaling action space that includes scaling to increased capacity in one of the network slices and scaling to reduced capacity in one of the network slices. In another aspect of the embodiment, the optimal result comprises a maximized inverse of the total network cost of the monitored result in the simultaneous time step.

In another embodiment of the invention, a C-RAN-architected data processing system can be adapted for continuous network slicing using delayed DDPG. The system includes a host computing platform built into a CU of a 5G network architecture for cellular communication. The system also includes a delayed DDPG based continuous network slicing module. The module contains computer program instructions that are activated during execution in the host computing platform in order to define at least two network slices in the CU, for which network slices a reinforced learning guideline is to be loaded which contains an actor guideline and a critic model. The computer program instructions further continuously identify a state of each of the network slices in different time steps, give the identified state of the enhanced learning policy and obtain e from the enhanced learning policy for a simultaneous one of the time steps, an output scaling operation.

The program instructions apply the output scaling process even further to the CU by assigning the various computing resources to the corresponding VNFs in the CU to one of the network slices and monitoring a resource cost result of the determined scaling process in the CU, while the monitored result has a predetermined optimal result is compared for the determined scaling process. The program instructions further determine a statistical Q value in the critic model based on a difference between the monitored result and the optimal result and output a gradient for each of the actor policies and the critic model taking into account the statistical Q value determined. Finally, the program instructions apply the calculated gradients in each case to the actor guideline or the critic model in order to use them in a next specific scaling process in a subsequent one of the time steps. Notably, the application of the calculated gradient for the actor policy occurs at a rate that is less frequent than an application of the calculated gradient for the critics model.

Additional aspects of the invention are set forth in part in the following description, and in part will be apparent from the description or may be learned through practice of the invention. The aspects according to the invention are realized and achieved by the elements and combinations particularly emphasized in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the invention as claimed.

Figure list

• ist eine schematische Darstellung eines C-RAN, der für verzögerte DDPG-basierte kontinuierliche Netzwerk-Slicing geeignet ist; und
• ist ein Flussdiagramm, das einen Prozess für C-RAN veranschaulicht, der für verzögerte DDPG-basierte kontinuierliche Netzwerk-Slicing konzipiert ist.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. The embodiments illustrated herein are presently preferred, it being understood, however, that the invention is not limited to the precise arrangements and instruments wherein

• Figure 3 is a schematic representation of a C-RAN suitable for DDPG-based delayed continuous network slicing; and
• Figure 13 is a flow diagram illustrating a process for C-RAN designed for DDPG-based delayed continuous network slicing.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention provide C-RAN designed for delayed DDPG-based continuous network slicing. The C-RAN comprises a host computing platform that is equipped in a CU of a 5G network with a cellular network. A DDPG-based continuous network slicing module runs in the platform's memory and defines two network slices in the CU while it is running. The module also loads a reinforced learning guideline for each of the network slices, which contains a simultaneous actor guideline, which takes the state of one of the network slices as input and delivers a specific scaling process as output, which provides the corresponding VNFs in the CU with different computing resources assigns a corresponding one of the network slices based on the identified status of the network. The enhanced learning policy also includes a critic model that takes the status of a corresponding one of the network slices in combination with the determined scaling operation as input and delivers a statistical Q-value as output.

As soon as the network slices are defined and the policies loaded, the module continuously identifies a status of each of the network slices in different time steps, provides the actor policy with the identified state and, for a simultaneous one of the time steps, denotes an output scaling process. The module then applies the output scaling operation to the CU by assigning the various computing resources to the appropriate VNFs in the CU to that of the network slices. The module then monitors a resource cost result of the determined scaling process in the CU and compares the monitored result with a predetermined optimal result for the determined scaling process.

• • Beschnittenes doppeltes Q-Learning mit zwei Kritikermodellen
• • Verzögerte Richtlinienaktualisierungen und Zielmodelle
• • Zielpolitikglättung und Lärmregularisierung

Finally, the module determines a statistical Q-value in the critic model based on a difference between the monitored result and the optimal result and provides a gradient for each of the actor's policies and the critic model, which takes the determined statistical Q-value into account. The module then applies the calculated gradient s to each of the actor guidelines and the critic model for use in a next particular scaling operation in a subsequent one of the time steps. It is important, however, that the module update the guideline for concurrent actors with the corresponding calculated gradient only at a rate that is less frequent than applying each calculated gradient to the critic model. In this way, a delayed DDPG-based continuous network slicing can be achieved, which leads to a considerable improvement in performance compared to a conventional dynamic, enhanced learning-based network slicing process. In particular, delayed DDPG addresses DDPG drawbacks by focusing on reducing the overestimation bias of previous algorithms by adding 3 key features:

• • Trimmed double Q-learning with two critic models
• • Delayed policy updates and target models
• • Target policy smoothing and noise regularization

In the further illustration is Figure 3 is a schematic representation of a C-RAN adapted for delayed DDPG-based continuous network slicing. As in shown, a C-RAN 130 be implemented to a host computing platform 100 include that one or more computers 110 each with main memory 140 and one or more processors 120 includes. In the storage room 140 are several different CUs 150 for the respective network slices 170 each defined one or more VNFs 160 to support the processing of 5G cellular network connections with different UEs 190 until you 180 support. It is important to have a delayed DDPG network slicing module 200 In the storage room 140 is included and from at least one of the processors 120 the host computing platform 100 is performed.

The delayed DDPG Network Slicing Module 200 contains computer program instructions that when executed in memory 140 a status for one of the network slices 170 received and the status of an actor guideline 115A provide. The Actor Policy 115 returns a scaling operation, including adding more CPUs 120 to one of the CUs 150 for the network slice 170 or removing one or more CPUs 120 from a corresponding one of the CUs 150 for the network slice 170 . The program instructions then monitor a result of the scaling process and compare the result with an optimal result. The comparison is made with the pair of critic models 135A , 145A provided together with the scaling operation, so that the critic models each generate a corresponding statistical Q value, which is then amalgamated with a minimization operation.

The program instructions then provide the amalgamated Q value in a gradient for each of the actor guidelines 115A and the criticism model 135A , 145A and the gradients for each of the actor policies 115A and the model's critics guidelines 135A , 145A are each aimed at a corresponding actor goal 115B and a pair of critic goals 135B applied. 145 B. Finally, the program instructions update the critic model p 135A , 145A with the critic goals 135B , 145B and the program instructions update the actor guideline 115A with the actor goal 115B , but the program instructions carry out the update of the actor guideline 115A at a rate that is less frequent than a rate of update of the critic model 135A . 145A . In this way, a delayed DDPG in the C-RAN 130 of the 5G network architecture cellular network.

was gleich $$E_s\sim p_{\pi }\sim a\sim {\pi }_{\varphi }\left[{\nabla }_{\varphi }{\pi }_{\varphi }{\left(a|s\right)|}_{a=\pi \left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]$$

As can be seen, the program instructions of the delayed DDPG network slicing module 200 aim at the parameters φ of the actor policy 115A to adapt in the direction of a power gradient v φJ (π _φ). The performance gradient that affects actor politics 115A applies can be reflected mathematically as follows: $${\nabla }_{\varphi }J\left({\pi }_{\varphi }\right)={\displaystyle \int {}_{\mathrm{S.}}{p}_{\pi }}\left(s\right){\displaystyle \int {}_{\mathrm{A.}}}{\nabla }_{\varphi }{\pi }_{\varphi }\left(a|s\right)Q^{\pi }\left(s,a\right)dads$$

what the same $${\mathrm{E.}}_s\sim p_{\pi }\sim a\sim {\pi }_{\varphi }\left[{\nabla }_{\varphi }{\pi }_{\varphi }{\left(a|s\right)|}_{a=\pi \left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]$$

The Actor Directive 115A can be parameterized as a value function with the aim of finding the optimal guideline πφ in which φ includes the updating of the weight of the actor policy 115A. The expected rate of return can be approximated in many ways. In one example, the gradient of the expected return _{can be calculated according to parameters of φ such as ∇ φ} J (φ). As can be seen, gradient descent is preferred over gradient descent in order to update the parameters, φ _{t + 1} = φ _t + α∇ _φ J (πφ) | φ _t .

wo $$Q^{\pi }\left(s,a\right)={\mathbb{E}}_{s\sim p_{\pi }\sim a_f\sim \pi }\left[R_t|s,a\right]$$

ist der statistische Q-Wert, der auch als Wertfunktion oder Kritiker bezeichnet wird.In the actor-critic method of two models work at the same time, with the actor guideline 115A is a rule that takes a policy state as input and provides actions as output while the s 135A , 145A merges respectively states and actions and returns the Q value, so that the actor guideline 115A can be updated by the deterministic policy gradient. $${\nabla }_{\varphi }J\left(\varphi \right)={\mathbb{E.}}_{s\sim p_{\pi }}\left[{\nabla }_a{Q}^{\pi }{\left(s,a\right)|}_{a=\pi \left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]$$

Where $$Q^{\pi }\left(s,a\right)={\mathbb{E.}}_{s\sim p_{\pi }\sim a_f\sim \pi }\left[{\mathrm{R.}}_t|s,a\right]$$

is the statistical Q-value, also known as the value function or critic.

In particular, a random experience is first stored in buffer-β. In other words, (s _t , a _t , r _t , s _t +1) is stored to train a Deep Q network. A random batch B is then selected in buffer β and for all transitions (s _tB , a _tB , r _tB , s _tB +1) of β, the predictions are Q (s _tB , a _tB ) and consider the goals to be optimal instantaneous Returns, which are exactly the first part of the learning error of the time difference (TD), as R (s _tB , a _tB ) + γmax _a (Q ( _{stB + 1} , a)). Over the entire batch B, the loss between predictions and the targets in batch B can be calculated. It is preferred to use a different target model instead of using the Q-network to compute the target in order to meet more stability for the learning algorithm. As can be seen, the TD process is based on the actor-critic model, while three additional processes are used to improve the TD algorithm:

(1) Truncated double Q-learning with two

The first additional process uses two Deep Neural Networks (DNNs) as the two actor models 115A , 115B and is used by φ as DNN for actor policy 115A and φ 'as DNN for the actor target 115B designated. In addition, there are two pairs of DNNs for critic models 135A , 135B and critic goals 135B , 145B provided and designated as θ1, θ2 for the parameterization of a value network as a critic model 135A , 145A and θ ' ₁ , θ' ₂ as a critic aims at 135B , 145B . Therefore, two machine learning processes take place at the same time, namely Q-Learning and Policy Learning, and the combination deals with demeaning errors, the reduction of bias and the definition of the highest statistical Q-value. For every element and transition of the batch, the actor objective plays a role 115B a 'based on s' while adding Gaussian noise to a'. The critic aims at 145A , 145B takes the pair (s ', a') and returns two Q values Q ' _t1 and Q' _t2 as output. Then the (min Q ' _t1 , Q' _t2 ) as the amalgamation of statistical Q values is used as the approximate value for DNNs of the critic's goals 145A , 145B considered.

The DNNs for the critic targets 145A , 145B are used to deliver the estimates by amalgamating the generated statistical Q-values as follows: $$Q_t=r+\gamma \ast \text{min}\left(Q_{t1}^i,Q_{t2}^i\right)$$

$${\nabla }_{\varphi }J\left(\varphi \right)=N^{-1}{\displaystyle \sum \left[{\nabla }_a{Q}_{\theta 1}{\left(s,a\right)|}_{a=\pi \left(\varphi \right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]}$$

Consequently, the DNNs give for the two critic models 135A , 135B returns two Q values as Q ₁ (s, a) and Q ₂ (s, a). The loss can then be based on the two models 135A , 135B and calculated with Mean Squared Error (MSE). To minimize the loss over iterations through back propagation technique, an efficient optimizer called Adaptive Moment Estimation can be used: $$\mathrm{L.}=l_{\mathrm{M.}\mathrm{S.}\mathrm{E.}}\left(Q_1,Q_t\right)+l_{\mathrm{M.}\mathrm{S.}\mathrm{E.}}\left(Q_2,Q_t\right)$$

$${\nabla }_{\varphi }J\left(\varphi \right)=N^{-1}{\displaystyle \sum \left[{\nabla }_a{Q}_{\theta 1}{\left(s,a\right)|}_{a=\pi \left(\varphi \right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]}$$

(2) Delayed policy updates and target models:

$${\varphi }^l\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^l$$

wobei τ ≤ 1 ein Hyperparameter ist, um die Aktualisierungsgeschwindigkeit zu optimieren.The second additional process sees a delayed update of the actor guideline 115A before. In particular, the DNN becomes the Actors Directive 115A updated less often than the DNNs for the critics models 135A , 135B to estimate values with less variance. The update rule is given by Polyak Averaging to update the parameters as follows: $${\theta }_f^l\leftarrow \tau {\theta }_f+\left(1-\tau \right){\theta }_f^l$$

$${\varphi }^l\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^l$$

where τ ≤ 1 is a hyperparameter to optimize the update speed.

(3) Target policy smoothing and noise regulation:

wobei das Rauschen aus einer Gaußschen Verteilung mit Null und einer bestimmten Standardabweichung abgetastet und in einem bestimmten Wertebereich zwischen -c und c abgeschnitten wird, um die Exploration zu fördern. εUmFehler bei der Verwendung eines unmöglichen Wertes von Aktionen zuvermeiden, wird das zusätzliche Rauschen auf den Bereich der möglichen Aktionen (min action, max_action) abgeschnitten.The third additional process serves the purpose of the actor's goal 115B and the critic target 135B , 145B to smooth. In this regard, when updating the critic model 135A , 145A a learning objective 135B , 145B using a deterministic guideline, very susceptible to inaccuracies caused by functional approximation errors, which increases the variance of the target. This induced variance is reduced by regularization to be sure that all possible continuous parameters can be explored. To do this, a 'Gaussian noise is added to the next action to prevent two large actions from perturbing the state of the environment: $$\tilde{a}\leftarrow {\pi }_{{\varphi }^l}\left(s^l\right)+\in ,\in \sim clip\left(N\left(\mathrm{0,}\tilde{\sigma }\right),-c,c\right)$$

wherein the noise from a Gaussian distribution with zero and a certain standard deviation is sampled and clipped in a certain range of values between -c and c in order to promote exploration. To avoid errors when using an impossible value of actions, the additional noise is cut off to the area of possible actions (min action, max_action).

The above TD3 -based network slicing method can be summarized as follows:

To further illustrate and summarize the above TD3-based network slicing methodology, Figure 2 is a flow diagram illustrating a process for C-RAN designed for delayed DDPG-based continuous network slicing. From block 210 a network slice is created in the CU as an environment and in block 215 initialized in memory for the network slice. In block 220 an actor policy and an actor target for a selected network segment are loaded into memory. Then be in block 225 also loaded two critic models and two critic targets. After that, in a block 230 , the actions are played randomly and in blocks 235 a stack of transitions is scanned. In block 240 the actor target takes the next state and plays a next action. In the block 245 Gaussian noise is added to the next action and the next action is pinched. In block 250 calculate the critic goals Q-values from the state and the action and in block 255 a minimum is calculated for the Q values. In block 260 a final target in terms of a discount factor is set and in block 265 the critic models receive the action and give and return Q-values. In block 270 will be the loss of critics in terms of the ultimate goal and block 275 calculated, the critic loss is propagated back to update the parameters of the critic models. In the decision block 280 it is determined whether the update of the actor model should be delayed. If so, will be in a block 285 the actor model updated with the output of the first critic model and in block 290 the weightings of each of the actor and critic goals are updated.

Thus, in accordance with the present invention, a reward penalty mechanism is provided to mitigate the adverse effects of destabilizing training. The reward-penalty mechanism cuts the network values to some constant and restrictive values with regard to the Quality of Service (QoS) and other threshold values. So, as will be seen, the proposed technique uses robotic algorithms in the telecommunications field. The repetition of experiences is also one the main aspects of learning behavior in biological systems. In order to accelerate the training process and improve the learning efficiency, a score-based asynchronous actor-learner is optimized for the network slicing environment.

The present invention can be embodied in a system, a method, a computer program product, or any combination thereof. The computer program product may include a computer readable storage medium or a medium containing computer readable program instructions for causing a processor to carry out aspects of the present invention. The computer readable storage medium can be a tangible device that can store and store instructions for use for an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.

Computer readable program instructions described herein can be downloaded from a computer readable storage medium or over a network to appropriate computing / processing devices. The computer-readable program instructions can run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer, and partially on a remote computer, or entirely on the remote computer or server. Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams and combinations of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer readable program instructions

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device for making a machine so that the instructions executed by the computer's processor or other programmable data processing device provide means for performing the steps in the flowchart and / or block diagram block or blocks provide specified functions / acts. These computer-readable program instructions can also be stored on a computer-readable storage medium, which can instruct a computer, a programmable data processing device and / or other devices to function in a certain way, so that the computer-readable storage medium on which instructions are stored, an article of manufacture including instructions that implement aspects of the function / action indicated in the flowchart and / or block diagram block or blocks.

The computer-readable program instructions can also be downloaded to a computer, other programmable data processing device, or other device to perform a series of operations on the computer, other programmable device, or other device to generate a computer-implemented process such that the instructions that running on the computer, other programmable devices; or another device implement the functions / actions indicated in the flowchart and / or block diagram block or blocks.

The flow diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products in accordance with various embodiments of the present invention. In this context, each block in the flowchart or block diagram can represent a module, segment, or part of instructions that contain one or more executable instructions for implementing the specified logical function (s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks that are displayed one after the other can actually be executed essentially simultaneously, or the blocks can sometimes be executed in reverse order, depending on the functionality involved. It is also pointed out that each block of the block diagrams and / or flowchart illustrations and combinations of blocks in the block diagrams and / or flowchart illustrations can be implemented by special hardware-based systems that perform or act on the specified functions or combinations of special hardware and follow computer instructions.

Finally, the terminology used herein is only used to describe particular embodiments and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is further understood that the terms "include", "includes" and / or "including" when used in this specification indicate the presence of specified features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or level-plus-functional elements in the following claims are intended to encompass any structure, material, or act for performing the function in combination with other claimed elements as expressly claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and its practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as particular contemplated Use are suitable.

Having thus described the invention of the present application in detail and with reference to the embodiments thereof, it will be clear that modifications and variations are possible without departing from the scope of the invention as defined in the appended claims as follows:

Claims (7)

A cloud radio data processing system (C-RAN) suitable for continuous network slicing, the system comprises: a host computing platform that has a central unit (CU) of a 5G network with a cellular network, the CU, which comprises a communicative coupling to a large number of different distributed units (DUs), at least one of the DUs, which has a massive MIMO antenna (Multiple Input, Multiple Output), which transmits over millimeter wave frequencies, the platform comprising one or more computers, each comprising memory and at least one processor; and a DDPG-based continuous network slicing module that includes computer program instructions that, while being executed, are enabled in the host computing platform to do the following: Defining at least two network slices in the CU; Load for the network slices a reinforced learning policy, which includes an actor policy that takes the state of one of the network slices as input and delivers a specific scaling process as output, which different computing resources the corresponding virtual network functions (VNFs) in the CU for one assigns the network slices, based on the identified state of one of the network slices, the enhanced learning policy, which additionally comprises a critic model that takes the state of the network slice in combination with the particular scaling process as input and a statistical Q-value returns as output; continuously at different times in order to identify a state of each of the network slices, to provide the identified state for the enhanced learning policy and to receive an output scaling process from the enhanced learning policy for a simultaneous one of the time steps; Application of the output scaling process to the CU by assigning the various computing resources to the corresponding VNFs in the CU to that of the network slices; monitoring of a resource cost result of the determined scaling process in the CU and comparison of the monitored result with a predefined optimal result for the determined scaling process Determination of a statistical Q value in the critic model based on a difference between the monitored result and the optimal result and calculation of a gradient for each of the actor policies and the critic model taking into account the determined statistical Q value; and applying the calculated gradients to each one of the actor policy and the critic model for use in a next particular scaling operation in a subsequent one of the time steps; but applying one of the calculated gradients that correspond to the actor's policy at a rate that is less frequent than applying the other of the calculated gradients to the critic model. System of Claim 1 , wherein the ascertained scaling process is determined taking into account a state space for that of the network slices, which a number of new user equipment connections (UE) to that of the network slices, computing resources that are assigned to each of the VNFs in the CU for that of the network segments, a delay status with respect to the latency costs for each of the at least two network segments, an energy status in relation to energy costs for the use of the computing resources by each of the at least two network slices, a number of users who are served in each of the at least two network slices, and a number of VNF instantiations in each of the at least two network slices. System of Claim 1 wherein the scaling process is part of a vertical scaling action space comprising scaling to increased capacity in one of the network slices and scaling to reduced capacity in one of the network slices. System of Claim 1 , wherein the optimal result comprises a maximized inverse of the total network cost of the monitored result while taking one of the time steps. The system of the Claim 1 , with the enhanced learning policy including twins of the critic model. System of Claim 5 , where the Q-value used in calculating the gradient for actor policy is a minimum of Q-values provided by each of the first twins. A computer program product for continuous network slicing, the computer program product including a computer readable storage medium having program instructions executable by a device to cause the device to perform a method, including: Definition of at least two network slices in a central unit (CU) of a 5G network architecture cellular network; Load for the network slices a reinforced learning policy, which includes an actor policy that takes the state of one of the network slices as input and delivers a specific scaling process as output, which different computing resources the corresponding virtual network functions (VNFs) in the CU for one assigns the network slices, based on the identified state of one of the network slices, the enhanced learning policy, which additionally includes a critic model that takes the state of the network slice in combination with the specific scaling process as input and a statistical Q - Returns value as output; continuously at different times in order to identify a state of each of the network slices, to provide the identified state for the enhanced learning policy and to receive an output scaling process from the enhanced learning policy for a simultaneous one of the time steps; Application of the output scaling process to the CU by assigning the various computing resources to the corresponding VNFs in the CU to that of the network slices; monitoring of a resource cost result of the determined scaling process in the CU and comparison of the monitored result with a predefined optimal result for the determined scaling process Determination of a statistical Q value in the critic model based on a difference between the monitored result and the optimal result and calculation of a gradient for each of the actor policies and the critic model taking into account the determined statistical Q value; and applying the calculated gradients to each one of the actor policy and the critic model for use in a next particular scaling operation in a subsequent one of the time steps; but applying one of the calculated gradients that correspond to the actor's policy at a rate that is less frequent than applying the other of the calculated gradients to the critic model.

Images