WO2023199098A1 - Safe exploration for automated network optimization using ml explainers - Google Patents

Abstract

Methods for evaluating and/or optimizing performance of a communication network. An example method comprises modifying each of one or more network configuration parameters for the communication network to a random offset from respective baseline values for the one or more network configuration parameters, where each random offset or modified network configuration parameter falls within a respective safety margin for the network configuration parameter. The method further comprises collecting performance data for one or more performance metrics of the network, after said modifying, and training a machine-learning model that infers said performance metrics from at least the one or more network configuration parameters, based on the modified network configuration parameters and the collected performance data. The method comprises repeating the modifying, collecting, and training steps multiple times.

Description

^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ SAFE EXPLORATION FOR AUTOMATED NETWORK OPTIMIZATION USING ML EXPLAINERS TECHNICAL FIELD This disclosure is generally related to communications networks and machine learning, ^^ and is more particularly related to techniques for improving network performance, using machine learning. BACKGROUND Wireless telecommunication networks are configured to run with best performance by using correct configuration management parameters, which may be set and/or tuned to ^^^ address varied cell sizes, deployment topographies, traffic load patterns, etc. Different configurations induce different behavior in the network, ideally to optimize network throughput, minimize interference and dropped/interrupted calls or data sessions, and otherwise optimize user experience. Normally, the parameters are configured based on guidelines provided by the respective network vendors for running the networks effectively. ^^^ Modern telecommunication networks – especially mobile networks - are extremely complex and this complexity is continuously increasing. Performance perceived by end users can be impacted by configuration, load and interactions of thousands of different networks elements. To keep operations cost low and end-user perceived performance high, it is critical to automate network operations and network optimization as much as possible. ^^^ There exist products performing data-driven closed-loop continuous optimization of network configuration parameters, with these products automatically tuning antenna tilts, load balancing configurations, etc. In recent years, machine-learning (ML) algorithms have been applied to the problem of optimizing communication network performance. Network data such as the configuration ^^^ management parameters discussed above, performance management metrics, and network events from event logs and alarms may be used as input parameters for machine- learning models. These ML models act as generalized models for the respective networks and can provide specific outputs for the given input data, based on the data patterns perceived by the algorithms. ^^^ ML methods require large amounts of data to learn from. In order to use machine learning to optimize high dimensional network configuration spaces, it is also important that this high dimensional space is well covered in the training data set. Unfortunately, this is ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ difficult to achieve for real networks, since the configuration is usually preferred to be kept as uniform and as stable as possible to avoid unwanted service degradations. In some other domains, i.e., outside the context of wireless communications networks, there exist easy-to-use data augmentation techniques to increase sample size and quality ^^ (coverage) of training data for ML algorithms. These techniques include such things as geometrical transformations, color manipulations, etc. which are applied to pictures in ML pipelines for image recognition. This is possible thanks to the invariance property of these transformations. However, this invariance property is unfortunately not applicable to the network optimization domain. ^^^ One solution to address this problem is the use of simulators, where arbitrary configurations can be emulated with real-world service degradations. However, simulators can never capture the full complexity of systems and workload in real networks. Reinforcement learning (RL) also involves exploration phases; hence it could be used in mobile networks to discover the state space while also converging towards a more optimal ^^^ state. The use of constrained RL can also ensure that exploration does not result in significant performance degradations. However, the complexity of RL methods has negative consequences with regards to the explainability of the solution. The concept of “Explainable AI” (xAI) has been introduced to provide additional insights, to the user of a system, about how the input parameters are influencing a particular decision. ^^^ As an example, the bar chart illustrated in Figure 1 indicates how each of various input parameters, which are “features” of a machine-learning model, are influencing the target parameter, which in this example is a power consumption metric. This sort of analysis is generally referred to as feature impact analysis. Because reliability and stability of communication network operation is critical, the reduced explainability resulting from the ^^^ use of RL makes operators reluctant to turn over control of the network to RL-based algorithms. U.S. Patent Application Publication No.2019/306023 A1 describes a system that monitors a network and receives data indicative of networking device configuration changes in the network. The system also receives one or more performance indicators for the network ^^^ and trains a machine learning model based on the received data indicative of the networking device configuration changes and on the received one or more performance indicators. The service then predicts, using the machine learning model, a change in the one or more performance indicators that would result from a particular networking device ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ configuration change. The system then causes the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators. Challenges remain, however, to ensuring that network configuration changes based on ^^ machine-learning processing do not threaten the reliable and stable operation of the network. SUMMARY Various embodiments of the techniques and systems described herein address the problems identified above by providing a safe state exploration solution for live ^^^ telecommunication networks. According to the techniques described herein, the baseline network configuration setup is periodically altered by small random offsets, within a thin safety margin. The resulting diverse configuration and workload data is used to train machine learning models to infer performance metrics that characterize the performance of the system. These supervised learning models are then used to continuously and ^^^ automatically optimize network configuration. Safety constraints for exploration (to prevent performance degradations from exceeding predefined small thresholds) are ensured using the explainers of these models quantifying the impact of small random offsets and extrapolating from them. These techniques can be also considered as data augmentation methods – similarly to ^^^ how geometrical transformations, color manipulations, etc., are applied to pictures in ML pipelines for image recognition. The difference is that in the systems discussed here, it is not possible to perform data augmentation directly on the collected data. Rather, small changes need to be applied continuously to the underlying system from where training data is being collected. ^^^ An example method for evaluating communication network performance, according to several of the presently disclosed techniques, comprises the step of modifying each of one or more network configuration parameters for the communication network to a random offset from respective baseline values for the one or more configuration parameters, wherein each random offset or modified network configuration parameter falls within a ^^^ respective safety margin for the network parameter. The method further comprises collecting performance data for one or more performance metrics of the network, after this modifying, and training a machine-learning model that infers the performance metrics from at least the one or more network configuration parameters, based on the modified network configuration parameters and the collected performance data. The method may comprise ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ repeating these modifying, collecting, and training steps multiple times, e.g., for a predetermined number of times, for a predetermined time period, or until certain performance goals are achieved. The proposed state exploration solution provides a training data set from live ^^ telecommunication networks where the high dimensional configuration space is well covered within the safety margin of each configuration parameter. This allows supervised learning methods to learn accurately how increasing or decreasing configuration values for specific parameters impact performance in this very complex system. In turn, learned good models allow highly reliable automated network configuration optimization. Meanwhile, thin ^^^ safety thresholds and feature impact analysis ensure that this state exploration process does not result in any significant performance degradations in the live networks. Furthermore, compared to reinforcement learning methods, the techniques described herein provide better explainability and lower risks for temporary performance degradation during the exploration process, thanks to the simplicity of this solution. Unlike most ^^^ reinforcement learning problems, optimization of telecommunication networks is mostly memoryless, therefore, it can be reduced to a combination of exploration and supervised learning, providing a simpler and more explainable solution. BRIEF DESCRIPTION OF THE FIGURES Figure 1 is an illustration of feature impact scores produced by explainable artificial ^^^ intelligence. Figure 2 illustrates an overview of the ML-based system described herein, according to some embodiments. Figure 3 is a flowchart illustrating periodic operation of a system according to some of the presently disclosed techniques. ^^^ Figure 4 is a process flow diagram illustrating an example method according to some of the presently disclosed techniques. Figure 5 is a block diagram showing an example processing node for carrying out one or more of the presently disclosed techniques. Figure 6 illustrates a virtualization environment, in which parts of or all of any of the ^^^ techniques disclosed herein may be implemented. ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ DETAILED DESCRIPTION Advantageous techniques for use in improving network performance are described herein. The network referred to herein can be a fifth generation (5G) wireless network, or any other communication network. In various embodiments, the network referred to herein may ^^ be a radio access network (RAN), or any other type of network. The techniques described may be implemented on or by one or more nodes, where those one or more nodes are part of or coupled to the network. The term “machine learning model” is well understood by those familiar with machine learning technology. Nevertheless, for the purposes of this document, the term should be ^^^ understood to mean a combination of a model data stored in machine memory and a machine-implemented predictive algorithm configured to infer one or more parameters, or “labels,” from one or more input data parameters, or “features.” Thus, a “machine learning model” is not an abstract concept, but an instantiation of a data structure comprising the model data coupled with an instantiation of the predictive algorithm. ^^^ Figure 2 illustrates elements of an example system employing ML-based system optimization techniques according to some embodiments of the present invention. The system comprises a database that contains a baseline configuration value, the current configuration parameter value, value constraints, and safety margins for each of several configuration parameters for one or more network element. The baseline configuration is a ^^^ baseline configuration value, i.e., a starting point for that configuration parameter. This might be, for example, an initial setting upon bringing up a new network element, or a manually tuned parameter value. The current configuration parameter value reflects any random offset(s) applied to the configuration parameter value, during the exploration process. ^^^ The value constraints are constraints for the values of respective configuration parameters. These value constraints may be specified in terms of a valid range for the configuration parameter or a set of discrete values for the configuration parameter. The value constraints may specify a minimum resolution for changes in the value of the configuration parameter. For an example, a parameter specifying a required minimum level of reference ^^^ symbol received power (RSRP) for cell reselection can be set in the range of -140 to -44 dBm, with a resolution of 2 dBm. These numbers represent the value constraints on this RSRP configuration parameter. ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ The safety margin for each configuration parameter specifies a minimum and maximum offset (or ratio) defining the range in which the configuration value can be altered around the baseline value as part of the exploration process. This safety margin is necessarily within the limits set by the value constraints for the configuration parameter, and may be ^^ considerably smaller, in some cases. These safety margins may be adjusted over time and may themselves be adapted by a ML-based algorithm. The database for the configuration parameters may further include timing parameters for each of one or more of the configuration parameters, with these timing parameters specifying one or more limits on the timing of random changes to the configuration ^^^ parameter(s) during the exploration process. These might be specified in terms of a minimum periodicity, e.g., every 15 minutes, but might also be specified in terms of a probability distribution for the change interval. These timing parameters may be selected to ensure sufficient settling of the system, in response to a change to the configuration parameter. ^^^ The database for the configuration parameters may still further contain offset distribution parameters for each of one or more of the configuration parameters, where the offset distribution parameters specify how to draw a random offset from the baseline configuration parameter, within the safety margin range. The system further comprises a database of one or more performance thresholds. These ^^^ are values representing the maximum absolute and/or relative degradation allowed for each of one or more monitored performance metrics for the system, which might be regarded as “guard metrics,” as a result of applying small random offsets to configuration parameters, starting from the baseline configuration values. For example, the performance thresholds may specify a maximum of 1% decrease in downlink throughput, and/or a max ^^^ 0.1% increase in drop rate. Separate thresholds might be specified for the individual impact of any given random offset or for the aggregate impact of all offsets. The configuration management component of the system shown in Figure 2 is responsible to apply small random changes continuously to each configuration parameter of each network element, subject to the various constraints, safety margins, timing parameters, ^^^ etc., specified in the database entries above. It will be appreciated that only configuration parameters with a range of settings that can be represented by a range of continuous numerical values or a set of discrete numerical values are supported by this system – the technique cannot generally be applied to ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ categorical variables. Examples for supported continuous configuration parameters in mobile networks are antenna tilt, antenna azimuth, transmission power, various signal strength or signal quality thresholds, etc. Examples for unsupported categorical variables are on/off flags, frequency bands, software versions, etc. ^^ The network monitoring component shown in Figure 2 continuously retrieves load metrics and other performance management (PM) metrics from each of one more monitored network elements, e.g., on a periodic basis that might be aligned, with an appropriate delay to allow for system settling, to the timing of the random adjustments to the configuration parameters. One or more of the performance metrics may be based on ^^^ measurements performed by user equipments (UEs) using the network and reporting the measurements to the network, in some embodiments. Retrieved PM data is correlated with CM data from the configuration management component, and is used in the ML system component to train one or more ML models that infer key performance metrics against which the network should be optimized. Optionally, once these ML models are trained, ^^^ baseline configurations for the managed network elements can be adjusted automatically to continuously improve network performance and adapt it to dynamically changing load conditions. Details for performing closed-loop optimization of the telecommunication network based on a trained ML algorithm are known, and are outside the scope of this invention. ^^^ In some embodiments, a feature impact analysis component can be attached to the ML system, as shown in Figure 2, to derive feature impact contributions. These can be used to automatically adjust safety margins via a safety logic component. Initially, this safety logic sets safety margins to a very narrow range, considering also the granularity/resolution of the given configuration parameter value settings from the value constraints database. ^^^ From feature impact analysis results, the safety logic component can compare the maximum impact of random feature offsets against the overall variability of performance metrics (both for individual features and aggregated for all CM features where random small offsets are applied). As long as these maximum impacts are significantly smaller than overall variability of performance metrics (also due to the impact of external factors ^^^ such as e.g., load, sub-cell location, etc.), the safety logic can widen the safety margin range for one or more of the configuration parameters. Optionally, this feature impact analysis component can also be used to evaluate changes made to the baseline configuration and separate the impact of these configuration changes from the impact of external factors, e.g., using techniques like those described in International Patent ^^^ Application Publication No. WO 2021/250445, the entire contents of which are ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ incorporated herein by reference. The feature impact analysis component may be implemented using machine learning model explainers, as described in that publication. For example, the feature impact analysis performed by the feature impact analysis component may comprise analyzing network performance metrics over time, in particular, ^^ one or more metrics, versus corresponding configuration management parameters, to identify contributions from changes to one or more of these configuration management to the one or more metrics. The performance metrics are predicted by the ML system component. In this way, it is possible to separate the impact of the changed configuration parameters from other factors, such as varying environmental and/or noise effects. This is ^^^ made possible by using an extended set of measurements that include measurements reflective of these other factors, in addition to measurements that are directly responsive to changes in the configuration parameter values. The person skilled in the art of machine- learning technologies will be aware of various techniques that can be used to identify the contribution of one or more features to the output of the ML system, i.e., to the one or ^^^ more performance metrics. One example is the Shapley additive explanations technique, which is well known for use in explaining the output of an ML model. Other explainable artificial intelligence (explainable AI) techniques might be used, instead of or in addition to the Shapely technique. The output of feature impact analysis according to these approaches might be feature ^^^ impact scores for each of several “features” of the ML system model, where these features include one or several configuration parameters. An example was shown in Figure 1. These scores can be used to identify which factors have the greatest effect on a given performance metric, or whether certain factors have any impact at all. Accordingly, it is possible to estimate the effect of the configuration changes on the ^^^ network, and separate those effects from other influencing factors, such as environmental disturbances, load, etc. As a practical matter, it can be good practice to align the periodicity of configuration changes to the periodicity of performance metric measurements. For example, if PM metrics are collected for fixed time periods every 15 minutes, then CM changes could also ^^^ be applied at the boundaries of these fixed periods, perhaps with the CM changes applied at a predetermined time offset prior to the PM metric collection, to allow for settling of the network after the change. For configuration changes having some serious impact on performance (for example, causing short outages, e.g., due to component restart ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ required), the change frequency could be lower (e.g., changing only once per day in low- traffic hours). The flow chart of Figure 3 illustrates the periodic operation of the proposed system. Initially, safety margins are filled in manually by domain experts (in a very conservative ^^ way) to ensure that performance impact of random configuration offsets remains minor (below the specified performance thresholds). Then at each iteration, a random offset is applied to baseline configuration parameters. After a period elapsed, performance measurements are evaluated using ML models and ML model explainers to derive the impact of each random offset. If this impact is much smaller than predefined thresholds for ^^^ performance metrics, than safety margin can be widened before the next iteration. However, if this impact exceeds the threshold, then the safety margin might be decreased. Otherwise, safety margins are left unchanged. Safety margin adjustment is performed separately for each configuration parameter. However, feature impact analysis and the application of random offsets can be synchronized and performed in the same batch ^^^ operation. In view of the detailed examples and explanation provided above, it will be appreciated that Figure 4 illustrates an example method for evaluating and/or improving communication network performance, according to at least some of the techniques that were detailed above. It should be appreciated that the illustrated method is intended to encompass several of the ^^^ techniques described above, and thus where there are minor differences in the terminology used to describe the method shown in Figure 4, this terminology should be understood as synonymous with or encompassing similar or related terms used in the preceding discussion. As shown at block 410, the method of Figure 4 includes the step of modifying each of one or ^^^ more network configuration parameters for the communication network to a random offset from respective baseline values for the one or more network configuration parameters, wherein each random offset or modified network configuration parameter falls within a respective safety margin for the network configuration parameter. The method further comprises, as shown at block 420, collecting performance data for one or more performance ^^^ metrics of the network, a predetermined period of time after said modifying. As shown at block 430, the method comprises training a machine-learning algorithm that infers values for the performance metrics, which might be referred to as “labels,” using machine-learning terminology, from at least the one or more network configuration parameters, or “features,” where this training is based on the modified network configuration parameters and the ^^^ collected performance data. The training, of course, may be further based on one or more ^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ other network parameters, in addition to the modified network configuration parameters and the collected performance data. The modifying, collecting, and training shown in blocks 410, 420, and 430 are repeated, multiple times. In some embodiments, this repeating of the modifying, collecting, and training steps is ^^ performed on a periodic basis. In some of these and in some other embodiments, the repeating is performed a predetermined number of times, or for a predetermined period of time. In others, the repeating is performed until each of one or more measured performance metrics reaches a respective predetermined goal. Note that the particular network configuration parameter or parameters that is/are modified may vary, from one iteration of ^^^ the method to another. In some embodiments, the method comprises selecting the random offset for each of the one or more network configuration parameters according to one or more offset distribution parameters for the network configuration parameter. This selecting is also subject to the safety margin for each parameter, which may fall within a validity range for the respective ^^^ network configuration parameter. In some embodiments, the method further comprises performing feature impact analysis evaluating an impact of changing one or more of the network configuration parameters on one or more performance metrics for the communication network, based on the trained machine-learning algorithm, and making an adjustment to one or more of the respective ^^^ safety margins, based on the feature impact analysis. This is shown at block 440. Note that while updates to the feature impact analysis might be made for each cycle of the steps shown in blocks 410, 420, and 430, in other embodiments this might be performed at longer intervals. In some instances, making the adjustment comprises reducing a safety margin for a first one of the network configuration parameters, based on determining that changes to ^^^ the first one of the network configuration parameters caused a degradation in a performance metric beyond a predetermined threshold degradation. In some of these and in other instances, making the adjustment may comprise increasing a safety margin for a second one of the network configuration parameters, based on determining that changes to the second one of the network configuration parameters did not cause a degradation in a performance ^^^ metric beyond a predetermined threshold degradation. Changes to multiple safety margins, corresponding to multiple network configuration parameters may be made at the same time, in some embodiments and/or instances. In some embodiments, the method may further comprise, subsequently to repeating the modifying, collecting, and training steps multiple times, using the trained machine-learning ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ algorithm to automatically adjust one or more of the network configuration parameters, to improve one or more performance metrics for the communication network. This is shown at block 450 of Figure 4. In some embodiments or instances, the exploration process illustrated in blocks 410-440 may be turned off at this point. In others, the exploration may continue, or ^^ be re-started after this optimization, with the adjusted network configuration parameters providing a new baseline configuration for subsequent exploration. Figure 5 illustrates an example processing node 500 in which all or parts of any of the techniques described above might be implemented. Processing node 500 may comprise various combinations of hardware and/or software, including a standalone server, a blade ^^^ server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. Processing node 500 may communicate with one or more radio access network (RAN) and/or core network nodes, in the context of a communications network, e.g., for collection of network performance data and/or for the monitoring and adjusting of network configuration parameters. ^^^ Processing node 500 includes processing circuitry 502 that is operatively coupled via a bus 504 to an input/output interface 506, a network interface 508, a power source 510, and a memory 512. Other components may be included in other embodiments. Memory 512 may include one or more computer programs including one or more application programs 514 and data 516. Embodiments of the processing node 500 may utilize only a ^^^ subset or all of the components shown. The application programs 514 may be implemented in a container-based architecture. It will be appreciated that multiple processing nodes may be utilized to carry out any of the techniques described herein, e.g., by allocating different functions to different nodes. Figure 9 illustrates several example cloud implementations. In a first example, for instance, various ^^^ functions of the CFR algorithms described herein are implemented as Function-as-a-Service (FaaS) functions deployed in a serverless FaaS system. This option of deployment can be for both cloud and near edge platforms, where functions are built with CFR as additional functionalities are available with them. In a second example, the CFR implementation is available as a side-car container with application. This option of deployment can be for both ^^^ cloud and near edge platform applications. Applications that prefer to do the life cycle management of CFR like it does for itself prefer this architecture. In a third option, CFR is available as pod with its own scaling and security. This option is the only option for edge devices to get CFR functionalities, as they are resource constrained. Also, this option is ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ available for near edge and cloud as alternative architecture where applications and functions prefer to use a common pod rather than having a side car container. Figure 6 is a block diagram illustrating a virtualization environment 600 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing ^^ means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be ^^^ implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 600 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely ^^^ virtualized. Applications 602 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein. ^^^ Hardware 604 includes processing circuitry, memory that stores software and/or instructions executable by hardware processing circuitry, and/or other hardware devices as described herein, such as a network interface, input/output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 606 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 608a and 608b ^^^ (one or more of which may be generally referred to as VMs 608), and/or perform any of the functions, features and/or benefits described in relation with some embodiments described herein. The virtualization layer 606 may present a virtual operating platform that appears like networking hardware to the VMs 608. The VMs 608 comprise virtual processing, virtual memory, virtual networking or interface and ^^^ virtual storage, and may be run by a corresponding virtualization layer 606. Different embodiments of the instance of a virtual appliance 602 may be implemented on one or more of VMs 608, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment. In the context of NFV, a VM 608 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of ^^ the VMs 608, and that part of hardware 604 that executes that VM, be it hardware dedicated to that VM and/or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 608 on top of the hardware 604 and corresponds to the application 602. ^^^ Hardware 604 may be implemented in a standalone network node with generic or specific components. Hardware 604 may implement some functions via virtualization. Alternatively, hardware 604 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 610, which, among others, oversees lifecycle management of applications 602. ^^^ The techniques, apparatuses, systems, and software-based solutions described above may be used to provide good quality training data for supervised learning methods in network optimization, covering well the multi-dimensional state space of network configuration. This training data collection is implemented with minimal impact on network performance during the exploration process thanks to continuous and automated impact evaluation. ^^^ Compared to reinforcement learning methods, the disclosed solutions provide better explainability and lower risks for temporary performance degradation during the exploration process, thanks to the simplicity of this solution. Unlike most reinforcement learning problems, optimization of telecommunication networks is mostly memoryless, therefore, it can be reduced to a combination of exploration + supervised learning, providing a simpler ^^^ and better explainable solution. The foregoing merely illustrates the principles of the disclosure. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements, and procedures that, although not explicitly shown ^^^ or described herein, embody the principles of the disclosure and can be thus within the spirit and scope of the disclosure. Various embodiments can be used together with one another, as well as interchangeably therewith, as should be understood by those having ordinary skill in the art. ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ The term unit or module, as used herein, can have conventional meaning in the field of electronics, electrical devices and/or electronic devices and can include, for example, electrical and/or electronic circuitry, devices, modules, processors, memories, logic solid state and/or discrete devices, computer programs or instructions for carrying out ^^ respective tasks, procedures, computations, outputs, and/or displaying functions, and so on, as such as those that are described herein. Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. ^^^ These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processor (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as Read Only Memory (ROM), ^^^ Random Access Memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and/or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional ^^^ unit to perform corresponding functions according one or more embodiments of the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should ^^^ be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. In addition, certain terms used in the present disclosure, including the specification and drawings, can be used synonymously in certain instances (e.g., “data” and “information”). ^^^ It should be understood, that although these terms (and/or other terms that can be synonymous to one another) can be used synonymously herein, there can be instances when such words can be intended to not be used synonymously. All publications referenced are incorporated herein by reference in their entireties. ^^^

Claims

^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ CLAIMS What is claimed is: 1. A method for evaluating performance of a communication network, the method comprising: modifying (410) each of one or more network configuration parameters for the communication network to a random offset from respective baseline values for the one or more network configuration parameters, wherein each random offset or modified network configuration parameter falls within a respective safety margin for the network configuration parameter; collecting (420) performance data for one or more performance metrics of the network, after said modifying; training (430) a machine-learning model that infers said performance metrics from at least the one or more network configuration parameters, wherein said training is based on the modified network configuration parameters and the collected performance data; and repeating said modifying, collecting, and training steps multiple times. 2. The method of claim 1, wherein said repeating is performed on a periodic basis. 3. The method of claim 1 or 2, wherein said repeating is performed a predetermined number of times, or for a predetermined period of time. 4. The method of claim 1 or 2, wherein said repeating is performed until each of one or more measured performance metrics reaches a respective predetermined goal. 5. The method of any one of claims 1-4, wherein said repeating comprises, for at least one of the multiple times, modifying a different one or more network configuration parameters compared to a previous one of the multiple times. 6. The method of any one of claims 1-5, wherein said training is further based on one or more additional network metrics other than the modified network configuration parameters. 7. The method of any one of claims 1-6, wherein the method comprises selecting the random offset for each of the one or more network configuration parameters according to one or more offset distribution parameters for the network configuration parameter. ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ 8. The method of any one of claims 1-7, wherein the safety margin for each of one or more of the network configuration parameters falls within a validity range for the respective network configuration parameter. 9. The method of any one of claims 1-8, wherein the method further comprises performing (440) feature impact analysis evaluating an impact of changing one or more of the network configuration parameters on one or more performance metrics for the communication network, based on the trained machine-learning model, and making an adjustment to one or more of the respective safety margins, based on the feature impact analysis. 10. The method of claim 9, wherein making the adjustment comprises reducing a safety margin for a first one of the network configuration parameters, based on determining that changes to the first one of the network configuration parameters caused a degradation in a performance metric beyond a predetermined threshold degradation. 11. The method of claim 9, wherein making the adjustment comprises increasing a safety margin for a second one of the network configuration parameters, based on determining that changes to the second one of the network configuration parameters did not cause a degradation in a performance metric beyond a predetermined threshold degradation. 12. The method of any of claims 1-11, wherein the method further comprises, subsequently to repeating the modifying, collecting, and training steps multiple times, using (450) the trained machine-learning model to automatically adjust the baseline value of one or more of the network configuration parameters, to improve one or more performance metrics for the communication network. 13. The method of any of claims 1-12, wherein one or more of the one or more performance metrics are based on measurements performed by user equipments, UEs, using the communication network and reported to the communication network. 14. One or more processing nodes (500) for use in or in association with a communication network, each of the one or more processing nodes (500) comprising processing circuitry (502) and a memory (512) operatively coupled to the processing circuitry and comprising program instructions for execution by the processing circuitry (502), whereby the processing nodes (500) are configured to: modify each of one or more network configuration parameters for the communication network to a random offset from respective baseline values for the one or ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ more network configuration parameters, wherein each random offset or modified network configuration parameter falls within a respective safety margin for the network configuration parameter; collect performance data for one or more performance metrics of the network, after said modifying; train a machine-learning model that infers said performance metrics from at least the one or more network configuration parameters, wherein said training is based on the modified network configuration parameters and the collected performance data; and repeat said modifying, collecting, and training steps multiple times. 15. The one or more processing nodes (500) of claim 14, wherein the processing nodes (500) are configured to carry out a method according to any of claims 2-13. 16. One or more processing nodes (500) for use in or in association with a communication network, the processing nodes (500) being adapted to carry out a method according to any of claims 1-13. 17. A computer program product comprising program instructions for execution by one or more processing circuits in one or more processing nodes operating in or in association with a communication network, wherein the program instructions are configured to cause the processing nodes to carry out a method according to any of claims 1-13. 18. A computer-readable medium comprising, stored thereupon, a computer program product according to claim 17. ^^^