EP4666219A1

EP4666219A1 - Machine learning model monitoring using multiple inferences

Info

Publication number: EP4666219A1
Application number: EP24706512.1A
Authority: EP
Inventors: Yufei Blankenship; Jung-Fu Cheng; Atieh Rajabi KHAMESI
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2023-02-17
Filing date: 2024-02-15
Publication date: 2025-12-24
Also published as: CN120660100A; WO2024171099A1

Abstract

A method by a radio node for monitoring a machine learning, ML, model, includes obtaining, for a number, n, of versions of an input sample at a point in time, t, a number, n, of versions of an output of the ML model The radio node measures a distribution of the disagreement between the number, n, of versions of the output over time. Based on the distribution of the disagreement between the number, n, of versions of the output over time, the radio node obtains at least one model monitoring metric.

Description

MACHINE LEARNING MODEL MONITORING USING MULTIPLE INFERENCES TECHNICAL FIELD The present disclosure relates, in general, to wireless communications and, more particularly, systems and methods for machine learning (ML) model monitoring using multiple inferences. BACKGROUND Artificial Intelligence (AI) and ML have been investigated as promising tools to optimize the design of air-interface in wireless communication networks in both academia and industry. Example use cases include using autoencoders for Channel State Information (CSI) compression to reduce the feedback overhead and improve channel prediction accuracy; using deep neural networks for classifying Line-of-Sight (LOS) and No-Line-of-Sight (NLOS) conditions to enhance the positioning accuracy; using reinforcement learning for beam selection at the network side and/or the User Equipment (UE) side to reduce signaling overhead and beam alignment latency; and using deep reinforcement learning to learn an optimal precoding policy for complex Multiple Input Multiple Output (MIMO) precoding problems. In 3^rd Generation Partnership Project (3GPP) New Radio (NR) standardization work, there was a new Release 18 study item on AI/ML for NR air interface in May 2022. This study item explores the benefits of augmenting the air-interface with features enabling improved support of AI/ML based algorithms for enhanced performance and/or reduced complexity/overhead. Through studying a few selected use cases (CSI feedback, beam management, and positioning), this System Information (SI) aims at laying the foundation for future air-interface use cases leveraging AI/ML techniques. Building an AI/ML model includes several development steps where the actual training of the AI model is just one step in a training pipeline. An important part in AI/ML developing is the AI/ML model lifecycle management. This is illustrated in FIGURE 1, which depicts training and inference pipelines and their interactions within a model lifecycle management procedure. The AI model lifecycle management typically consists of ^ A training (re-training) pipeline, o With data ingestion referring to gathering raw (training) data from a data storage. After data ingestion, there may also be a step that controls the validity of the gathered data. o With data pre-processing referring to some feature engineering applied to the gathered data. For example, it may include data normalization and possibly a data transformation required for the input data to the AI/ML model. o With the actual model training steps. o With model evaluation referring to benchmarking the performance to some baseline. The iterative steps of model training and model evaluation continues until the acceptable level of performance (as previously exemplified) is achieved. o With model registration referring to register the AI/ML model, including any corresponding AI/ML-meta data that provides information on how the AI/ML model was developed, and possibly AI/ML model evaluations performance outcomes. ^ A deployment stage to make the trained (or re-trained) AI/ML model part of the inference pipeline. ^ An inference pipeline, o With data ingestion referring to gathering raw (inference) data from a data storage. o With data pre-processing stage that is typically identical to corresponding processing that occurs in the training pipeline. o With model operational referring to using the trained and deployed model in an operational mode. o With data & model monitoring referring to validate that the inference data are from a distribution that aligns well with the training data, as well as monitoring model outputs for detecting any performance, or operational, drifts. ^ A drift detection stage that informs about any drifts in the model operations. In a first scenario, it is assumed that AI/ML models operating with the existing standard air-interface are placed at the UE side. A UE uses the AI/ML models to generate output that is reported to a centralized node in the network for determining positioning of the UE location. In a second scenario, it is assumed that AI/ML models operating with the existing standard air-interface are placed at different Transmit/Receive Points (TRPs). A TRP uses the AI/ML models to generate output that is reported to a centralized node in the network for determining positioning of the UE location. There currently exist certain challenge(s), however. For example, existing methods for model monitoring generally belong to two categories. The first category is performance-based and includes examining certain quality metrics of the model output (e.g., inference accuracy, error rate, and/or system performance). The second category is data distribution-based and includes monitoring whether the model input distribution has drifted or whether the model output distribution has drifted. Performance-based methods are more reliable, easier to implement, and less prone to the problem of false alarm. However, performance-based methods typically assume that a ground truth label is somehow made available during model operation so that the ground truth label can be compared with the model output during deployment to determine whether the deployed model is still working properly. However, it is rarely the case that the ground truth label can be obtained during model deployment, and even if possible, it typically incurs much delay and overhead. With supervised methods, existing solutions require the system (either the UE or the network) to acquire a second position estimate from a different source to compare with the primary positioning estimate. For instance, for UE ML-model for positioning, the UE may use another positioning method to verify the positioning accuracy of the ML model from time to time. In one example, positioning-based on non-RAT information can be used if the UE has access to non-RAT signal such as GNSS, WiFi, Bluetooth, camera, and sensor. The UE can, for example, trigger a request to perform a UE ML-model verification upon retrieving its position via a non-RAT signal GNSS, WiFi, Bluetooth, camera, and/or sensor. However, such a second source position estimate may not always be feasible. For instance, GNSS cannot be used in indoor environments. WiFi and Bluetooth may not be used in sensitive industrial environments to avoid interference or eavesdropping. Furthermore, even when a second source position estimate is available, it is not clear which of the primary and second position estimates is less accurate. In fact, if the second position estimate is assumed to be always more accurate than the primary position estimate, then the system should discard the primary position estimation solution and adopt only the second position estimation solution. In unsupervised or statistical methods, existing solutions can detect if a potential change in the system has occurred but, in some cases, the change does not translate in a loss of performance in the monitored model. Thus, triggering retraining solely on this information can end-up being wasteful. SUMMARY Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. For example, methods and systems are provided for monitoring the performance of AI/ML models by generating multiple versions of model output. According to certain embodiments, a method by a radio node for monitoring a ML model includes obtaining, for an input sample at a point in time, t, a number, n, of versions of an output of the ML model. The radio node measures a distribution of the disagreement between the number, n, of versions of the output over time. Based on the distribution of the disagreement between the number, n, of versions of the output over time, the radio node obtains at least one model monitoring metric. According to certain embodiments, a radio node for monitoring a ML model is configured to obtain, for an input sample at a point in time, t, a number, n, of versions of an output of the ML model. The radio node is configured to measure a distribution of the disagreement between the number, n, of versions of the output over time. Based on the distribution of the disagreement between the number, n, of versions of the output over time, the radio node is configured to obtain at least one model monitoring metric. Certain embodiments may provide one or more of the following technical advantage(s). For example, certain embodiments may provide a technical advantage of enabling the same entity that performs inference to also monitor the performance of AI/ML model. Thus, other entities do not have to be relied upon. As another example, certain embodiments may provide a technical advantage of enabling monitoring of the AI/ML model to be deployed at network side or UE-side. As still another example, certain embodiments may provide a technical advantage of supporting various AI/ML functionalities, e.g., positioning, beam management, mobility management, link adaption, etc. As yet another example, certain embodiments may provide a technical advantage of providing a reliable method for determining whether the operating environment is still consistent with the characteristics of the environment from which training data was collected for training the AI/ML models. Certain embodiments may also provide a technical advantage of not requiring an external source to provide the ground truth of model output during model operation. Other advantages may be readily apparent to one having skill in the art. Certain embodiments may have none, some, or all of the recited advantages. BRIEF DESCRIPTION OF THE DRAWINGS For a more complete understanding of the disclosed embodiments and their features and advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which: FIGURE 1 illustrates training and inference pipelines within a model lifecycle management procedure; FIGURE 2 illustrates the low disagreements among the multiple model inferences when there is no model drift, and the high disagreements among the multiple model inferences when the model drift has occurred, according to certain embodiments; FIGURE 3 illustrates a trained ML model for deployment, according to certain embodiments; FIGURE 4 illustrates an exemplary model inference and model monitoring functions, according to certain embodiments; FIGURE 5 illustrates the AI/ML-based positioning example to illustrate the detection of drift region, according to certain embodiments; FIGURE 6 illustrates a summarization of the CDF of the test results, according to certain embodiments; FIGURE 7 illustrates a direct path ToA estimate L2 difference using unmodified or modified positioning request data at production in different operating environments for a small centralized assisted positioning model, according to certain embodiments; FIGURE 8 illustrates an example communication system, according to certain embodiments; FIGURE 9 illustrates an example UE, according to certain embodiments; FIGURE 10 illustrates an example network node, according to certain embodiments; FIGURE 11 illustrates a block diagram of a host, according to certain embodiments; FIGURE 12 illustrates a virtualization environment in which functions implemented by some embodiments may be virtualized, according to certain embodiments; FIGURE 13 illustrates a host communicating via a network node with a UE over a partially wireless connection, according to certain embodiments; FIGURE 14 illustrates an example method monitoring a ML model, according to certain embodiments; and FIGURE 15 illustrates a method by radio node for monitoring a ML model, according to certain embodiments. DETAILED DESCRIPTION Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. As used herein, ‘node’ can be a network node or a UE. Examples of network nodes are NodeB, base station (BS), multi-standard radio (MSR) radio node such as MSR BS, eNodeB (eNB), gNodeB (gNB), Master eNB (MeNB), Secondary eNB (SeNB), integrated access backhaul (IAB) node, network controller, radio network controller (RNC), base station controller (BSC), relay, donor node controlling relay, base transceiver station (BTS), Central Unit (e.g. in a gNB), Distributed Unit (e.g. in a gNB), Baseband Unit, Centralized Baseband, C-RAN, access point (AP), transmission points, transmission nodes, Remote Radio Unit (RRU), Remote Radio Head (RRH), nodes in distributed antenna system (DAS), core network node (e.g. Mobile Switching Center (MSC), Mobility Management Entity (MME), etc.), Operations & Maintenance (O&M), Operations Support System (OSS), Self Organizing Network (SON), positioning node (e.g. E- SMLC), etc. Another example of a node is user equipment (UE), which is a non-limiting term and refers to any type of wireless device communicating with a network node and/or with another UE in a cellular or mobile communication system. Examples of UE are target device, device to device (D2D) UE, vehicular to vehicular (V2V), machine type UE, MTC UE or UE capable of machine to machine (M2M) communication, Personal Digital Assistant (PDA), Tablet, mobile terminals, smart phone, laptop embedded equipment (LEE), laptop mounted equipment (LME), Unified Serial Bus (USB) dongles, etc. In some embodiments, generic terminology, “radio network node” or simply “network node (NW node),” is used. It can be any kind of network node which may comprise base station, radio base station, base transceiver station, base station controller, network controller, evolved Node B (eNB), Node B, gNodeB (gNB), relay node, access point, radio access point, Remote Radio Unit (RRU) Remote Radio Head (RRH), Central Unit (e.g. in a gNB), Distributed Unit (e.g. in a gNB), Baseband Unit, Centralized Baseband, C-RAN, access point (AP), etc. The term radio access technology (RAT), may refer to any RAT such as, for example, Universal Terrestrial Radio Access Network (UTRA), Evolved Universal Terrestrial Radio Access Network (E-UTRA), narrow band internet of things (NB-IoT), WiFi, Bluetooth, next generation RAT, NR, 4G, 5G, etc. Any of the equipment denoted by the terms node, network node or radio network node may be capable of supporting a single or multiple RATs. The term signal or radio signal used herein can be any physical signal or physical channel. Examples of DL physical signals are reference signal (RS) such as Primary Synchronization Signal (PSS), Secondary Synchronization Signal (SSS), Channel State Information-Reference Signal (CSI-RS), Demodulation Reference Signal (DMRS) signals in SS/PBCH block (SSB), discovery reference signal (DRS), Cell-specific Reference Signal (CRS), Positioning Reference Signal (PRS), etc. RS may be periodic. For example, RS occasions carrying one or more RSs may occur with certain periodicity such as, for example, 20 ms, 40 ms, etc. The RS may also be aperiodic. Each SSB carries NR-PSS, NR-SSS and NR-PBCH in 4 successive symbols. One or multiple SSBs are transmit in one SSB burst that is repeated with certain periodicity such as, for example, 5 ms, 10 ms, 20 ms, 40 ms, 80 ms, and 160 ms. The UE is configured with information about SSB on cells of certain carrier frequency by one or more SS/PBCH block measurement timing configuration (SMTC) configurations. The SMTC configuration comprising parameters such as SMTC periodicity, SMTC occasion length in time or duration, SMTC time offset with regard to reference time (e.g., serving cell’s SFN), etc. Therefore, SMTC occasion may also occur with certain periodicity such as, for example, 5 ms, 10 ms, 20 ms, 40 ms, 80 ms, and 160 ms. Examples of uplink (UL) physical signals are reference signal such as Sounding Reference Signal (SRS), DMRS, etc. The term physical channel refers to any channel carrying higher layer information, including data, control, etc. Examples of physical channels are Physical Broadcast Channel (PBCH), Narrowband PBCH (NPBCH), Physical Downlink Control Channel (PDCCH), Physical Downlink Shared Channel (PDSCH), Physical Uplink Shared Channel (PUSCH), Physical Uplink Control Channel (PUCCH), shortened PUCCH (sPUCCH), shortened PDSCH (sPDSCH), shortened PUCCH (sPUCCH), shortened PUSCH (sPUSCH), MTC PDCCH (MPDCCH), Narrowband PDCCH (NPDCCH), Narrowband PDSCH (NPDSCH), Enhanced-PDCCH (E- PDCCH), Narrowband PUSCH (NPUSCH), etc. The term time resource used herein may correspond to any type of physical resource or radio resource expressed in terms of length of time. Examples of time resources are: symbol, time slot, subframe, radio frame, Transmission Time Interval (TTI), interleaving time, slot, sub-slot, mini-slot, system frame number (SFN) cycle, hyper-SFN (H-SFN) cycle, etc. ML model drift refers to the phenomenon that a model's performance with input data at the model deployment stage is different from the expected performance when the model was trained with a training dataset. Model drift may occur due to various reasons. Typical reasons include, for example, poor training data and changing environments. While the issue of poor training data can be remedied with careful data collection, changing environments are not avoidable and can’t be controlled. Changing environments may cause the change in model input data statistics, and the relationship between model input data and model output data. The training dataset is composed of collected historical data, which carries inherent relationships about the data for the model to learn. When the model is deployed in a dynamically changing environment in the real word, the statistical properties of the data encountered at model deployment may shift from those of the stored training dataset over time. This may lead to the problem that the model can no longer produce accurate output. Without model adaptation, the ML model in production slowly deteriorates over time. Therefore, there is a need to continuously perform model monitoring during deployment. A drift indicator is triggered if model drift is detected according to certain criteria, indicating unexpected or unacceptable model performance. Then appropriate reactions can be taken when a drift indicator is received, for example, re-training the model or deactivating the model. According to certain embodiments, methods are provided to perform model monitoring by examining disagreements between model outputs. Certain embodiments leverage the redundancies that are widely present in a large ML model, where multiple versions of model output can be generated for a given model input. By examining the statistics of the multiple output versions, a model monitoring metric is calculated. If model drift is absent (i.e., the model is working properly), the disagreements between the two or more versions of model output follows one distribution (e.g., consistently low disagreements). On the other hand, if model drift is present, the disagreements between the two or more versions of model output may exhibit another distribution (e.g., high frequency of large disagreements). According to certain embodiments, for example, methods and systems are provided for monitoring the performance of AI/ML models by generating multiple versions of model output. For a given input sample ^^ _^^ at time t, multiple model inferences are performed to obtain ^^ versions ^{of model output: [ ^^ ^^(0), … , ^^ ^^( ^^ − 1)], ^^ ≥ 2. By measuring the distribution of [ ^^ ^^(0), … , ^^ ^^( ^^ −} _{1)] over time, model monitoring metrics are obtained to determine if the model is working} properly according to the design goals. In a particular embodiment, when ^^ = 2 , the distances between two different model outputs are checked against a threshold over time. In another particular embodiment, time series analysis of the different model outputs distribution over time can be used to compare with each other to detect a drift. FIGURE 2 illustrates an example 100 of low disagreements among multiple model inferences when there is no model drift, and the high disagreements among the multiple model inferences when the model drift has occurred, according to certain embodiments. Thus, the magnitude of the disagreements can be calculated as a measure of the model drift. FIGURE 3 illustrates an example 200 of a trained ML model for deployment, according to certain embodiments. Specifically, FIGURE 3 shows the model inference using the deployed ML ^{model M, with a length-K model input ^^ ^^ = [ ^^ ^^,0, ^^ ^^,1, … ^^ ^^, ^^−1] and a length-L model output ^^ ^^ =} _{[ ^^ ^^,0, ^^ ^^,1, … ^^ ^^, ^^−1]. Note this is a generic description input and output dimensions to} show that the input or output consists of multiple elements. Each element may itself be a scalar, vector, or multi-dimensional tensor. For instance, each element of the input may be a scaler representing the received power at a different of antenna ports or base station. As another example, each element of the input may be a vector representing the received channel impulse responses at different tap positions at a different of antenna ports or base station. For the purpose of performing model monitoring, for a given input sample ^^ _^^ at time t, ^{multiple model inferences are performed to obtain ^^ versions of model output: [ ^^ ^^(0), … , ^^ ^^( ^^ −} _{1)], ^^ ≥ 2. By checking the disagreements of the distribution of ^^ ^^(0), … , ^^ ^^( ^^ − 1) over time,} model monitoring metric ^^ _^^ can be generated. While in general, monitoring can be performed using two or more versions of model output, for brevity the discussion below assumes two versions of model output for model monitoring, i.e., ^^ = 2. It is understood by those skilled in the art that the same methodology can be easily extended to the cases with ^^ ≥ 3. FIGURE 4 illustrates an example 300 of model inference and model monitoring functions, according to certain embodiments. Xt(1) is a variant of Xt(0), where Xt(0)=Xt, i.e., the same as model input. ML model M ^ is a variant of the deployed model M. It is possible that X_t(1)=X_t(0) or M ^=M, but not at the same time so that Yt(1) is a different version than Yt(0). The disclosed methods have the following benefits: ^ The model monitoring metric can be generated by the model inference entity itself. No signaling overhead is needed. In contrast, performance-based model monitoring methods need another entity to provide feedback about the quality of the model output or the ground truth label corresponding to the model output, thus incurring signaling overhead. ^ The monitoring can be performed at the same time as model inference, thus minimizing detection delay. In contrast, performance-based model monitoring methods need to wait for another entity to signal feedback about the quality of the model output or the ground truth label corresponding to the model output, thus incurring longer detection delay. Methods to Obtain Multiple Inference Outputs According to certain embodiments, the multiple inference outputs can be obtained by applying a small perturbation to the normal model inference procedure. In a particular embodiment, a small perturbation can be applied to the model input without changing the ML model M. Further details are explained in the first subsection below. In another particular embodiment, a small perturbation can be applied to the ML model M in operation, without changing the model input. Further details are explained in the second subsection below. In yet another particular embodiment, a small perturbation can be applied to both the model input and the ML model M in operation at the same time. It may be understood by those skilled in the art that such combination can be easily applied with the teachings below. Multiple Inferences by Using Multiple Versions of Model Input According to certain embodiments, the ^^ versions of model output can be obtained by modifying the model input, while keeping the ML model M unchanged. A multitude of modifications, generally known as data augmentation, can be applied to generate modified inputs. A few widely used modification approaches are described below to illustrate the method. However, it should be clear to one skilled in the art that other data augmentation approaches can be applied to the embodiments disclosed herein. In one example embodiment, ^^ _^^ ⁽1⁾ is obtained from ^^ _^^ ⁽0⁾ by dropping a subset of the ^{elements in ^^ ^^(0) . For instance, ^^ ^^(0) = ^^ ^^ = [ ^^ ^^,0, ^^ ^^,1, … ^^ ^^, ^^−1] , and ^^ ^^(1) =} _{[ ^^ ^^,0, 0, ^^ ^^,2, 0, ^^ ^^,4, … , ^^ ^^, ^^−1] by dropping ^^ ^^,1 and ^^ ^^,3.} ^^ _^^(1) is obtained from ^^ _^^(0) by setting a subset of the elements in ^^ _^^ ⁽0⁾ to random noise values. For instance, ^^ _^^ ⁽1⁾ = [ ^^ _^^,0, ^^₀, ^^ _^^,2, ^^₁, ^^ _^^,4, … , ^^ _{^^, ^^−1}] by changing ^^ _^^,1 and ^^ _^^,3 to random noise values ^^₀ and ^^₁. In a further example embodiment, ^^ _^^(1) is obtained from ^^ _^^(0) by mixing it with the previous input data ^^ _^^−1(0): ^^ _^^(1) = ^^ ^^ _^^(0) + (1 − ^^) ^^ _^^−1(0) where 0 ≤ ^^ ≤ 1 is a random number. The random mixing may also be applied a randomly selected subset of the elements. In yet another example embodiment, ^^ _^^(1) is obtained from ^^ _^^(0) by adding noises: ^^ _^^(1) = ^^ _^^(0) + ^^ where ^^ is a length- ^^ random noises. For instance, Gaussian noises can be added. In another example embodiment, ^^ _^^(1) is obtained from ^^ _^^(0) by applying random time or frequency shifts to different elements. In another example embodiment, ^^ _^^ ⁽1⁾ is obtained from ^^ _^^ ⁽0⁾ by changing the resolution of its numerical representation, for example through down sampling or coarser quantizing. Multiple Inferences by Using Multiple Versions of the Model Structure According to certain embodiments, the ^^ versions of model output can be obtained by slightly modifying the ML model M, while keeping the model input ^^ _^^ unchanged. A multitude of modifications, generally known as model regularization, can be applied to generate modified models. A few widely used modification approaches are described below. However, it should be clear to one skilled in the art that other mode regularization approaches can be applied to the teaching disclosed herein. In one example embodiment, the model M is modified to a slightly different model M’ by dropping one or more layers. The layer to drop can be fixed or predefined. Alternatively, the layer to drop can be randomly selected layers. In another example embodiment, the model M is modified to a slightly different model M’ by setting some model parameters to zero, i.e. to skip some connections in the original model M. The modified parameters (weights and/or biases) can be fixed or predefined. Alternatively, the modified parameters are randomly selected weights and/or biases. In another example embodiment, the model M is modified to a slightly different model M’ by using a different activation function at certain predefined, or randomly selected, layer(s). For example, changing a sigmoid function to a RELU function. In another example embodiment, the model M is modified to a slightly different model M’ by setting different hyperparameters. Obtain the Monitoring Metrics Using the Multiple Inferences ^{With a pair of model output ^^ ^^(0) = [ ^^ ^^,0(0), ^^ ^^,1(0), … ^^ ^^, ^^−1(0)] and ^^ ^^(1) =} _{[ ^^ ^^,0(1), ^^ ^^,1(1), … ^^ ^^, ^^−1(1)], the distance between them over time can be calculated. Specifically,} for a length-T time window from time ^^ _^^ to ^^ _{^^+ ^^−1}, the distance between a first version of the ^{model output} {_{^^ ^^ ^^(0), ^^ ^^ ^^+1(0), ^^ ^^ ^^+2(0), … , ^^ ^^ ^^+ ^^−1(0)}} ^{and a second version of the} ^_{^ ^^ , ^^ , ^^ , … , ^^ ^^+ ^^−1 }} is calculated, where the metric. The size T of time window for the various metrics including drift detection accuracy, detection delay, and storage size of reference set and test set. In a particular embodiment, the p-norm distance is used. For a given time ^^ _^^ in the size-T time window, ^^ _^^ ≤ ^^ _^^ ≤ ^^ _{^^+ ^^−1}, the distance is calculated by: ^^ _{^^ ^^} = ‖ ^^ _{^^ ^^}(0) − ^^ _{^^ ^^}(1)‖ _^^ When ^^ = 1, the distance is the taxicab norm, ^^ _{^^ ^^} = ∑ ^{^} _^ ^{^} _^ ⁻ ₌ ¹ ₀ | ^^ _{^^ ^^, ^^}(0) − ^^ _{^^ ^^, ^^}(1)| . When ^^ = 2, the distance is the Euclidean norm, ^^ _{^^ ^^} = _√∑ ^{^} ^^{^} ^⁻ =¹ 0 ( ^^ _{^^ ^^, ^^}(0) − ^^ _{^^ ^^, ^^}(1))². When ^^ = ∞, the distance is the maximum norm, ^^ _{^^ ^^} = _^^= m_0,…ax_{, ^^−1} | ^^ _{^^ ^^, ^^}(0) − ^^ _{^^ ^^, ^^}(1)| To detect model drift, two sets of distances are obtained: the first set is the reference set, the second set is the test set (i.e., to be tested against the reference set). Model monitoring is obtained by comparing the statistical distribution between the two sets of distances. If the two sets of distances share the same or similar distribution according to a criterion, then the model monitoring function determines that no model drift is detected. On the other hand, if the two sets of distances exhibit different distribution, then the model monitoring function determines that model drift has been detected. ^ The reference set { ^^ _^ ^{^} ^^{^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^} ^_^ , ^^ _{^^ ^^+1} , … , ^^ _{^^ ^^+ ^^−1} } is typically obtained from the training dataset. set { ^^ _^ ^{^} ^^{^ ^^ ^^} ^_^ , ^^ _^ ^{^} ^^{^ ^^ ^^ ^^ ^^ ^^} ^_^+1 , … , ^^ _{^^ ^^+ ^^−1} } is obtained from an earlier time window of model deployment, for example, a historical period where the deployed model was working properly. If the data that the model works on has seasonality (e.g., a pattern that repeats approximately daily/weekly/monthly/annually/etc.), the reference time window should be chosen to contain data of similar distribution as the test time window, e.g., same hour of the day, same day of the week, etc. ^ The test set is the set of distances that is under examination by the model monitoring function, { ^^ _{^^ ^^} , ^^ _{^^ ^^+1} , … , ^^ _{^^ ^^+ ^^−1}}. The model monitoring metric of the test set against the reference set can be obtained by various statistical methods. Without losing the generality, several representative statistical methods are shown below. o Difference of mean value of the reference set and test set. For example, monitoring metric ^^ _{^^ ^^ ^^ ^^} = | ^^ _{^^ ^^ ^^} − ^^ _{^^ ^^ ^^ ^^}| is calculated and compared with a threshold. Let the threshold be ^^ × ^^ _{^^ ^^ ^^}. Then model drift is detected if | ^^ _{^^ ^^ ^^} − ^^ _{^^ ^^ ^^ ^^}| > ^^ × ^^ _{^^ ^^ ^^}; otherwise, no drift is detected. Here ^^ _{^^ ^^ ^^} and ^^ _{^^ ^^ ^^} are the mean value and standard deviation of the reference set { ^^ _^ ^{^} ^^{^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^} ^_^ , ^^ _{^^ ^^+1} , … , ^^ _{^^ ^^+ ^^−1} } , respectively. ^^ _{^^ ^^ ^^ ^^} is the mean value of the test set { ^^ _{^^ ^^} , ^^ _{^^ ^^+1} , … , ^^ _{^^ ^^+ ^^−1}}. ^^ is a parameter selected for the desired model monitoring sensitivity. A value can be ^^ = 3. o The difference at a given Cumulative Distribution Function (CDF) (e.g., 80% or 90%) between the reference set and test set. o Kolmogorov–Smirnov statistic, which finds the distance between the empirical distribution functions of the reference set and the test set. o The Population Stability Index (PSI), which is calculated using the formula below. Here the ^^ _{^^ ^^ ^^ ^^}( ^^) refers to the probability that the test set data is in bin ^^, and ^^ _{^^ ^^ ^^}( ^^) refers to the probability that the reference set data is in bin ^^. Parameter ^^ refers to the number of bins, and its value is a design choice. The value of parameter ^^ can be selected by taking into consideration of time window size T, the desired model detection sensitivity, and other considerations. ^^{^} ^{^^ ^^ ^^ ^^ ^^ ^^)} ) A^{model monitoring metric signal ^^ ^^ may be triggered based on the examination of test set} _{{ ^^ ^^ ^^ , ^^ ^^ ^^+1, … , ^^ ^^ ^^+ ^^−1}. Using the PSI metric as an example, the model monitoring metric generator} may produce three levels of output ^^ _^^. o If PSI < 0.1: no significant distribution difference between the reference set and test set. No alert signal is triggered, or: signal ^^ _^^=NULL. o PSI < 0.2: moderate distribution difference between the reference set and test set is detected. A warning signal is triggered, or: signal ^^ _^^=WARNING. o PSI >= 0.2: significant distribution difference between the reference set and test set is detected. An alarm signal is triggered, or: signal ^^ _^^=ALARM. It is understood by those in the art that, in general, the model monitoring metric generator may produce W levels of output ^^ _^^. Other than the W=3 example illustrated above, the model monitoring metric generator may use a two level signal (W=2) where ^^ _^^=NULL indicates “no model drift detected”, and ^^ _^^=ALARM indicates “model drift detected”. In addition to (or in place of) providing a W-level alert signal ^^ _^^, the model monitoring metric generator may also produce a drift tracking output, which can be called drift measurement. The drift measurement keeps a numerical record of how much the rest set deviates from the reference set over time. This follows an observability approach that tracks the magnitude of the drift without setting alert thresholds (e.g., 0.1 and 0.2 in the PSI example above). The decision of whether/when/how to react to the drift can be taken by another function which is fed with the drift measurement report. The drift measurement report is provided by model monitoring metric generator, either in a periodical manner or in a triggered manner. The drift measurement report may contain the desired numerical metric, e.g., ^^ _{^^ ^^ ^^ ^^}, PSI, as discussed above. In another nonlimiting embodiment, the distance metric ^^ _{^^ ^^} is compared to a threshold during model inference and monitoring operations. Said threshold can be determined from the statistical distributions of the reference set. Said threshold can be selected such that model drift is declared when said distance metric ^^ _{^^ ^^} is greater than said threshold. For convenience of description, in the above, it was assumed that the same window size T is used for both the reference set and the test set. However, the drift detection metric calculation is statistical in nature. It is not necessary that the reference set and the test set use the same window size, as long as the window size is sufficiently long to extract useful statistics. In actual deployment, it may be preferable to use a longer window size for the reference set in order to obtain accurate statistical estimation, whereas a shorter window size is preferred for the test set in order to have a shorter drift detection latency. Also, for convenience the teaching above assumed that samples are taken continuously from time ^^ _^^ to ^^ _{^^+ ^^−1} in the time window. It is understood that other variations are easily without deviating from the principle of the teaching. For instance, the samples are taken once every k instances; or only samples satisfying a certain criteria are taken in the calculation of the model monitoring metric. In addition, the procedure described above is for teaching and illustration purpose. In actual operation, efficient implementation of the procedure can be formulated without deviating from the principle. For example, the reference set does not have to be stored as is. It reduces storage overhead to extract and store only the desired statistics (e.g., mean, deviation, Probability Density Function (PDF), and/or CDF) of the reference set instead. Detection of Drift Region Following the method of generating multiple inferences using multiple versions of model input, the method can be carefully designed so that the drift detector can identify the sub-area of drift. In one nonlimiting embodiment, the model can be trained with a partition of the feature set so that the drift detector can identify the subset of features that caused the drift. Using the AI/ML-based positioning as an example, if the model has been trained to be robust to channel dropping, then dropping a fraction of the channel measurement is not expected to drastically degrade performance if no model drift is present. In the illustrated example below, thirty-six channel measurements are gathered as the input to the AI/ML model, with two measurements for each TRP due to cross-polarization of the receive antenna. If the model monitoring algorithm detected model drift when using the modified model input [0, ^^ _^^,1, … , ^^ _^^,5, 0, ^^ _^^,7, 0, ^^ _^^,8, … , ^^ _^^,35] (i.e., dropping ^^ _^^,0, ^^ _^^,6, ^^ _^^,8), then a hypothesis can be taken that the part of the model covering the colored region is no longer effective. The UEs in the problematic area cannot be located accurately anymore. Thus, one can isolate that area for model adaptation. In contrast, the model covering the rest of the deployment area can still be considered acceptable. FIGURE 5 illustrates an AI/ML-based positioning example 400 depicting the detection of drift region, according to certain embodiments. Application to the Positioning Use Case In the following, the disclosed method is applied to the positioning use case to demonstrate its effectiveness. Most AI/ML based positioning method exhibit sensitivity to environmental changes such as model drift. This is the case for any fingerprinting type of AI/ML models, both AI/ML-assisted positioning methods and direct AI/ML positioning methods. Since the fingerprinting type AI/ML models are sensitive to operating environment changes, it is necessary to monitor the model performance over time and ensure the models are operating within performance requirements. In general, model performance monitoring requires periodically obtaining additional new test samples with both the required model inputs and the correct UE positions. When models are trained with data augmentation, self-model monitoring with the disclosed method is possible without collecting new test samples with ground truth UE positions. Specifically, the model-based self-monitoring method is described below. ^ Model training with data augmentation o Given ground truth samples ( ^^ _^^, ^^ _^^) o Train the model with a training dataset which comprise a mixture of: ^ Unmodified samples ( ^^ _^^(0), ^^ _^^) , where ^^ _^^(0) = ^^ _^^ ^ Data augmentation ^^ ^^ , where ^^ _^^(1) is a modified version of ^^ _^^. Note that the modified sample share the same output ^^ _^^ as the unmodified sample. ^ Self-model monitoring at operation o Given positioning request data ^^ _^^ o Obtain a first target estimate ^^ _^^(0) using unmodified input data ^^ _^^(0), i.e., ^^ _^^ ⁽0⁾ = ^^ _^^. Note: ^^ _^^(0) is also used as the model inference output, i.e., ^^ _^^ = ^^ _^^(0) o second target estimate ^^ _^^(1) using data augmentation modified input data ^^ _^^(1), o Check the difference between the two target estimates, ^^ _^^ = ‖ ^^ _^^(0) − ^^ _^^(1) ‖. The effectiveness of this self-monitoring approach was investigated using a small model (0.73 M parameters) for the centralized direct positioning approach trained with InF-DH {60%, 6m, 2m} dataset samples. Here InF-DH refers to the channel model: Indoor Factory with Dense clutter and High base station height (Tx or Rx elevated above the clutter). FIGURE 6 illustrates a summarization 500 of the CDF of the test results, according to certain embodiments. Specifically, FIGURE 6 illustrates a 2D position estimate difference using unmodified or modified positioning request data at production in different operating environments for a small centralized direct positioning model trained with {60%, 6m, 2m} InF-DH dataset samples. It can be observed that: ^ When the model is operating in environments similar to that it was trained for (such as the {60%, 6m, 2m} and {40%, 6m, 2m} InF-DH environments), the two UE position estimates do not differ by more than 1 m. ^ When the model is operating in environments for which it was not trained and, thus, is under-performing, the two UE position estimates differ by more than 1 m with substantially probabilities. For instance, in the {60%, 2m, 2m} and {40%, 2m, 2m} environments, the two UE position estimates differ by more than 1 m with 20% and 33% probabilities, respectively. o While observing one UE position estimate difference of below 1 m is not reliable enough to ensure the model is still operating in the intended environment, checking the ^^ _^^ distribution in a small window of 10s-20s model inferences can achieve monitoring accuracy >99%. ^ Observing fourteen UE position estimate differences of below 1 m is enough to ensure an environment change from {60%, 6m, 2m} to {60%, 2m, 2m} is identified with a probability > 95%. ^ Observing twenty-one UE position estimate differences of below 1 m is enough to ensure an environment change from {60%, 6m, 2m} to {60%, 2m, 2m} is identified with a probability > 99%. ^ Observing eight UE position estimate differences of below 1 m is enough to ensure an environment change from {60%, 6m, 2m} to {40%, 2m, 2m} is identified with a probability > 96%. ^ Observing twelve UE position estimate differences of below 1 m is enough to ensure an environment change from {60%, 6m, 2m} to {40%, 2m, 2m} is identified with a probability > 99%. This investigation demonstrates that the distribution of ^^ _^^ can be used to generate model monitoring metrics. A variety of commonly used metrics for ML model monitoring can be considered, such as Kolmogorov–Smirnov (KS) test, Kullback–Leibler (KL) divergence, Population Stability Index (PSI). The self-model monitoring approach for models trained with data augmentation can also be applied to the AI/ML assisted positioning approaches. Instead of checking the differences between UE position estimates, the Layer 1 (L1) or Layer 2 (L2) differences between vectors of direct path Time of Arrival (ToA) estimates can be checked to identify environment changes. FIGURE 7 illustrates an example 600 of a direct path ToA estimate L2 difference using unmodified or modified positioning request data at production in different operating environments for a small centralized assisted positioning model trained with {60%, 6m, 2m} InF-DH dataset samples. Considering that data augmentation is widely used in model training, the self-model monitoring approach described above is a useful technique. Since the model monitoring can be performed together with model inference without any additional assistance (e.g., ground truth label) from other nodes during operation, it has the advantage of no requirement of obtaining ground truth label during operation, no signaling overhead, and low latency for the model monitoring function. FIGURE 8 shows an example of a communication system 700 in accordance with some embodiments. In the example, the communication system 700 includes a telecommunication network 702 that includes an access network 704, such as a radio access network (RAN), and a core network 706, which includes one or more core network nodes 708. The access network 704 includes one or more access network nodes, such as network nodes 710a and 710b (one or more of which may be generally referred to as network nodes 710), or any other similar 3^rd Generation Partnership Project (3GPP) access node or non-3GPP access point. The network nodes 710 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 712a, 712b, 712c, and 712d (one or more of which may be generally referred to as UEs 712) to the core network 706 over one or more wireless connections. Example wireless communications over a wireless connection include transmitting and/or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and/or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 700 may include any number of wired or wireless networks, network nodes, UEs, and/or any other components or systems that may facilitate or participate in the communication of data and/or signals whether via wired or wireless connections. The communication system 700 may include and/or interface with any type of communication, telecommunication, data, cellular, radio network, and/or other similar type of system. The UEs 712 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and/or operable to communicate wirelessly with the network nodes 710 and other communication devices. Similarly, the network nodes 710 are arranged, capable, configured, and/or operable to communicate directly or indirectly with the UEs 712 and/or with other network nodes or equipment in the telecommunication network 702 to enable and/or provide network access, such as wireless network access, and/or to perform other functions, such as administration in the telecommunication network 702. In the depicted example, the core network 706 connects the network nodes 710 to one or more hosts, such as host 716. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 706 includes one more core network nodes (e.g., core network node 708) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and/or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 708. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and/or a User Plane Function (UPF). The host 716 may be under the ownership or control of a service provider other than an operator or provider of the access network 704 and/or the telecommunication network 702, and may be operated by the service provider or on behalf of the service provider. The host 716 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio/video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server. As a whole, the communication system 700 of FIGURE 8 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and/or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and/or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and/or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox. In some examples, the telecommunication network 702 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 702 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 702. For example, the telecommunications network 702 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and/or Massive Machine Type Communication (mMTC)/Massive IoT services to yet further UEs. In some examples, the UEs 712 are configured to transmit and/or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 704 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 704. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio – Dual Connectivity (EN-DC). In the example, the hub 714 communicates with the access network 704 to facilitate indirect communication between one or more UEs (e.g., UE 712c and/or 712d) and network nodes (e.g., network node 710b). In some examples, the hub 714 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 714 may be a broadband router enabling access to the core network 706 for the UEs. As another example, the hub 714 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 710, or by executable code, script, process, or other instructions in the hub 714. As another example, the hub 714 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 714 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 714 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 714 then provides to the UE either directly, after performing local processing, and/or after adding additional local content. In still another example, the hub 714 acts as a proxy server or orchestrator for the UEs, in particular in if one or more of the UEs are low energy IoT devices. The hub 714 may have a constant/persistent or intermittent connection to the network node 710b. The hub 714 may also allow for a different communication scheme and/or schedule between the hub 714 and UEs (e.g., UE 712c and/or 712d), and between the hub 714 and the core network 706. In other examples, the hub 714 is connected to the core network 706 and/or one or more UEs via a wired connection. Moreover, the hub 714 may be configured to connect to an M2M service provider over the access network 704 and/or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 710 while still connected via the hub 714 via a wired or wireless connection. In some embodiments, the hub 714 may be a dedicated hub – that is, a hub whose primary function is to route communications to/from the UEs from/to the network node 710b. In other embodiments, the hub 714 may be a non- dedicated hub – that is, a device which is capable of operating to route communications between the UEs and network node 710b, but which is additionally capable of operating as a communication start and/or end point for certain data channels. FIGURE 9 shows a UE 800 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and/or operable to communicate wirelessly with network nodes and/or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded/integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and/or an enhanced MTC (eMTC) UE. A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and/or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter). The UE 800 includes processing circuitry 802 that is operatively coupled via a bus 804 to an input/output interface 806, a power source 808, a memory 810, a communication interface 812, and/or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in FIGURE 9. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc. The processing circuitry 802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 810. The processing circuitry 802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 802 may include multiple central processing units (CPUs). In the example, the input/output interface 806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and/or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 800. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device. In some embodiments, the power source 808 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 808 may further include power circuitry for delivering power from the power source 808 itself, and/or an external power source, to the various parts of the UE 800 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 808. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 808 to make the power suitable for the respective components of the UE 800 to which power is supplied. The memory 810 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 810 includes one or more application programs 814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 816. The memory 810 may store, for use by the UE 800, any of a variety of various operating systems or combinations of operating systems. The memory 810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and/or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 810 may allow the UE 800 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 810, which may be or comprise a device-readable storage medium. The processing circuitry 802 may be configured to communicate with an access network or other network using the communication interface 812. The communication interface 812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 822. The communication interface 812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 818 and/or a receiver 820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 818 and receiver 820 may be coupled to one or more antennas (e.g., antenna 822) and may share circuit components, software or firmware, or alternatively be implemented separately. In the illustrated embodiment, communication functions of the communication interface 812 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and/or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol/internet protocol (TCP/IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth. Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 812, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient). As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input. A UE, when in the form of an Internet of Things (IoT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an IoT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door/window sensor, a flood/moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item- tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an IoT device comprises circuitry and/or software in dependence of the intended application of the IoT device in addition to other components as described in relation to the UE 800 shown in FIGURE 9. As yet another specific example, in an IoT scenario, a UE may represent a machine or other device that performs monitoring and/or measurements, and transmits the results of such monitoring and/or measurements to another UE and/or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and/or reporting on its operational status or other functions associated with its operation. In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and/or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators. FIGURE 10 shows a network node 900 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and/or operable to communicate directly or indirectly with a UE and/or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)). Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units and/or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS). Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell/multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and/or Minimization of Drive Tests (MDTs). The network node 900 includes a processing circuitry 902, a memory 904, a communication interface 906, and a power source 908. The network node 900 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 900 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 900 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 904 for different RATs) and some components may be reused (e.g., a same antenna 910 may be shared by different RATs). The network node 900 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 900, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 900. The processing circuitry 902 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and/or encoded logic operable to provide, either alone or in conjunction with other network node 900 components, such as the memory 904, to provide network node 900 functionality. In some embodiments, the processing circuitry 902 includes a system on a chip (SOC). In some embodiments, the processing circuitry 902 includes one or more of radio frequency (RF) transceiver circuitry 912 and baseband processing circuitry 914. In some embodiments, the radio frequency (RF) transceiver circuitry 912 and the baseband processing circuitry 914 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 912 and baseband processing circuitry 914 may be on the same chip or set of chips, boards, or units. The memory 904 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and/or any other volatile or non-volatile, non-transitory device-readable and/or computer-executable memory devices that store information, data, and/or instructions that may be used by the processing circuitry 902. The memory 904 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and/or other instructions capable of being executed by the processing circuitry 902 and utilized by the network node 900. The memory 904 may be used to store any calculations made by the processing circuitry 902 and/or any data received via the communication interface 906. In some embodiments, the processing circuitry 902 and memory 904 is integrated. The communication interface 906 is used in wired or wireless communication of signaling and/or data between a network node, access network, and/or UE. As illustrated, the communication interface 906 comprises port(s)/terminal(s) 916 to send and receive data, for example to and from a network over a wired connection. The communication interface 906 also includes radio front- end circuitry 918 that may be coupled to, or in certain embodiments a part of, the antenna 910. Radio front-end circuitry 918 comprises filters 920 and amplifiers 922. The radio front-end circuitry 918 may be connected to an antenna 910 and processing circuitry 902. The radio front- end circuitry may be configured to condition signals communicated between antenna 910 and processing circuitry 902. The radio front-end circuitry 918 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 918 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 920 and/or amplifiers 922. The radio signal may then be transmitted via the antenna 910. Similarly, when receiving data, the antenna 910 may collect radio signals which are then converted into digital data by the radio front-end circuitry 918. The digital data may be passed to the processing circuitry 902. In other embodiments, the communication interface may comprise different components and/or different combinations of components. In certain alternative embodiments, the network node 900 does not include separate radio front-end circuitry 918, instead, the processing circuitry 902 includes radio front-end circuitry and is connected to the antenna 910. Similarly, in some embodiments, all or some of the RF transceiver circuitry 912 is part of the communication interface 906. In still other embodiments, the communication interface 906 includes one or more ports or terminals 916, the radio front-end circuitry 918, and the RF transceiver circuitry 912, as part of a radio unit (not shown), and the communication interface 906 communicates with the baseband processing circuitry 914, which is part of a digital unit (not shown). The antenna 910 may include one or more antennas, or antenna arrays, configured to send and/or receive wireless signals. The antenna 910 may be coupled to the radio front-end circuitry 918 and may be any type of antenna capable of transmitting and receiving data and/or signals wirelessly. In certain embodiments, the antenna 910 is separate from the network node 900 and connectable to the network node 900 through an interface or port. The antenna 910, communication interface 906, and/or the processing circuitry 902 may be configured to perform any receiving operations and/or certain obtaining operations described herein as being performed by the network node. Any information, data and/or signals may be received from a UE, another network node and/or any other network equipment. Similarly, the antenna 910, the communication interface 906, and/or the processing circuitry 902 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and/or signals may be transmitted to a UE, another network node and/or any other network equipment. The power source 908 provides power to the various components of network node 900 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 908 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 900 with power for performing the functionality described herein. For example, the network node 900 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 908. As a further example, the power source 908 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail. Embodiments of the network node 900 may include additional components beyond those shown in FIGURE 10 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and/or any functionality necessary to support the subject matter described herein. For example, the network node 900 may include user interface equipment to allow input of information into the network node 900 and to allow output of information from the network node 900. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 900. FIGURE 11 is a block diagram of a host 1000, which may be an embodiment of the host 716 of FIGURE 8, according to certain embodiments. As used herein, the host 1000 may be or comprise various combinations hardware and/or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 1000 may provide one or more services to one or more UEs. The host 1000 includes processing circuitry 1002 that is operatively coupled via a bus 1004 to an input/output interface 1006, a network interface 1008, a power source 1010, and a memory 1012. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 8 and 9, such that the descriptions thereof are generally applicable to the corresponding components of host 1000. The memory 1012 may include one or more computer programs including one or more host application programs 1014 and data 1016, which may include user data, e.g., data generated by a UE for the host 1000 or data generated by the host 1000 for a UE. Embodiments of the host 1000 may utilize only a subset or all of the components shown. The host application programs 1014 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1014 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1000 may select and/or indicate a different host for over-the-top services for a UE. The host application programs 1014 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc. FIGURE 12 is a block diagram illustrating a virtualization environment 1100 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1100 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. Applications 1102 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein. Hardware 1104 includes processing circuitry, memory that stores software and/or instructions executable by hardware processing circuitry, and/or other hardware devices as described herein, such as a network interface, input/output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1106 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1108a and 1108b (one or more of which may be generally referred to as VMs 1108), and/or perform any of the functions, features and/or benefits described in relation with some embodiments described herein. The virtualization layer 1106 may present a virtual operating platform that appears like networking hardware to the VMs 1108. The VMs 1108 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1106. Different embodiments of the instance of a virtual appliance 1102 may be implemented on one or more of VMs 1108, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment. In the context of NFV, a VM 1108 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1108, and that part of hardware 1104 that executes that VM, be it hardware dedicated to that VM and/or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1108 on top of the hardware 1104 and corresponds to the application 1102. Hardware 1104 may be implemented in a standalone network node with generic or specific components. Hardware 1104 may implement some functions via virtualization. Alternatively, hardware 1104 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1110, which, among others, oversees lifecycle management of applications 1102. In some embodiments, hardware 1104 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1112 which may alternatively be used for communication between hardware nodes and radio units. FIGURE 13 shows a communication diagram of a host 1202 communicating via a network node 1204 with a UE 1206 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 712a of FIGURE 8 and/or UE 800 of FIGURE 9), network node (such as network node 710a of FIGURE 8 and/or network node 900 of FIGURE 10), and host (such as host 716 of FIGURE 8 and/or host 1000 of FIGURE 11) discussed in the preceding paragraphs will now be described with reference to FIGURE 13. Like host 1000, embodiments of host 1202 include hardware, such as a communication interface, processing circuitry, and memory. The host 1202 also includes software, which is stored in or accessible by the host 1202 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1206 connecting via an over-the-top (OTT) connection 1250 extending between the UE 1206 and host 1202. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1250. The network node 1204 includes hardware enabling it to communicate with the host 1202 and UE 1206. The connection 1260 may be direct or pass through a core network (like core network 706 of FIGURE 8) and/or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet. The UE 1206 includes hardware and software, which is stored in or accessible by UE 1206 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1206 with the support of the host 1202. In the host 1202, an executing host application may communicate with the executing client application via the OTT connection 1250 terminating at the UE 1206 and host 1202. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1250 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1250. The OTT connection 1250 may extend via a connection 1260 between the host 1202 and the network node 1204 and via a wireless connection 1270 between the network node 1204 and the UE 1206 to provide the connection between the host 1202 and the UE 1206. The connection 1260 and wireless connection 1270, over which the OTT connection 1250 may be provided, have been drawn abstractly to illustrate the communication between the host 1202 and the UE 1206 via the network node 1204, without explicit reference to any intermediary devices and the precise routing of messages via these devices. As an example of transmitting data via the OTT connection 1250, in step 1208, the host 1202 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1206. In other embodiments, the user data is associated with a UE 1206 that shares data with the host 1202 without explicit human interaction. In step 1210, the host 1202 initiates a transmission carrying the user data towards the UE 1206. The host 1202 may initiate the transmission responsive to a request transmitted by the UE 1206. The request may be caused by human interaction with the UE 1206 or by operation of the client application executing on the UE 1206. The transmission may pass via the network node 1204, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1212, the network node 1204 transmits to the UE 1206 the user data that was carried in the transmission that the host 1202 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1214, the UE 1206 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1206 associated with the host application executed by the host 1202. In some examples, the UE 1206 executes a client application which provides user data to the host 1202. The user data may be provided in reaction or response to the data received from the host 1202. Accordingly, in step 1216, the UE 1206 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input/output interface of the UE 1206. Regardless of the specific manner in which the user data was provided, the UE 1206 initiates, in step 1218, transmission of the user data towards the host 1202 via the network node 1204. In step 1220, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1204 receives user data from the UE 1206 and initiates transmission of the received user data towards the host 1202. In step 1222, the host 1202 receives the user data carried in the transmission initiated by the UE 1206. One or more of the various embodiments improve the performance of OTT services provided to the UE 1206 using the OTT connection 1250, in which the wireless connection 1270 forms the last segment. More precisely, the teachings of these embodiments may improve one or more of, for example, data rate, latency, and/or power consumption and, thereby, provide benefits such as, for example, reduced user waiting time, relaxed restriction on file size, improved content resolution, better responsiveness, and/or extended battery lifetime. In an example scenario, factory status information may be collected and analyzed by the host 1202. As another example, the host 1202 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1202 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1202 may store surveillance video uploaded by a UE. As another example, the host 1202 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1202 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and/or transmitting data. In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1250 between the host 1202 and UE 1206, in response to variations in the measurement results. The measurement procedure and/or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1202 and/or UE 1206. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1250 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1250 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1204. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1202. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1250 while monitoring propagation times, errors, etc. FIGURE 14 illustrates an example method monitoring a ML model, according to certain embodiments. The method may be performed by a wireless device or a network node in various particular embodiments. In the illustrated embodiment, the method includes an obtaining step at 1302, a measuring step at 1304, and an obtaining step at 1306. For example, at step 1302, the UE and/or the network node may obtain a number, n, of versions of an output of the ML model for an input sample at a point in time, t. At step 1304, the UE and/or the network node may measure a distribution of the number, n, of versions of the output over time. At step 1306, based on the distribution, the UE and/or the network node may obtain at least one model monitoring metric. FIGURE 15 illustrates a method 1400 by a radio node for monitoring a ML model, according to certain embodiments. As illustrated the method begins at step 1402 when, for an input sample at a point in time, t, the radio node obtains a number, n, of versions of an output of the ML model. At step 1404, the radio node measures a distribution of the disagreement between the number, n, of versions of the output over time. Based on the distribution of the disagreement between the number, n, of versions of the output over time, the radio node obtains at least one model monitoring metric. In a particular embodiment, the radio node is a UE 712. In a particular embodiment, the radio node is a network node 710. In a particular embodiment, when obtaining the n versions of an output of the ML model, the radio node performs model inference n times. In a further particular embodiment, when performing the model inference n times, the radio node obtains n versions of the input sample and uses a different version of the input sample each time the model inference is performed. In a further particular embodiment, the input sample includes a set of elements. When obtaining the n versions, the radio node performs data augmentation on the set of elements. In a further particular embodiment, when performing data augmentation the radio node performs at least one of: dropping a subset of a set of elements in the input sample; setting a subset of the set of elements in the input sample to random noise values; mixing the input sample with previous input data; adding noise to the input sample; applying a random time or frequency shift to elements in the input sample; and changing a resolution of the input sample. In a further particular embodiment, when performing the model inference n times, the radio node obtains n versions of the ML model and uses a different version of the ML model each time the model inference is performed. In a further particular embodiment, when obtaining the n versions of the ML model, the radio node uses and/or performs model regularization to generate the n versions of the ML model. In a particular embodiment, when using and/or performing model regularization, the radio node performs at least one of: dropping one or more layers; setting at least one model parameter to zero; using a different activation function on at least one layer; adding noise to the input sample; and using and/or setting different hyperparameters. In a particular embodiment, n is greater than or equal to 2, and when obtaining the at least one model monitoring metric the radio node determines a magnitude of disagreement between two of the n versions of the output for a length, T, time window. In a particular embodiment, the radio node compares the magnitude of disagreement between the two of the n versions of the output to a threshold. In a particular embodiment, the radio node compares the magnitude of disagreement between the two of the n versions of the output to reference magnitudes of disagreement associated with a reference set. In a particular embodiment, the radio node determines model drift based on a difference between the magnitude of disagreement between the two of the n versions of the output and the reference magnitudes of disagreement. In a particular embodiment, the reference set is based on historical data obtained when the ML model was known to be operating properly. In a particular embodiment, the magnitude of disagreement includes a distance between two of the n versions of the output. In a particular embodiment, T is determined based on a type of the at least one model monitoring metric. In a particular embodiment, when obtaining the at least one model monitoring metric the radio node uses a statistical method to determine the at least one model monitoring metric. The statistical method comprises at least one of: a difference of mean value; a difference at a given Cumulative Distribution Function, CDF; a Kolmogorov statistic method; and a population stability index. In a particular embodiment, the radio node performs a time series analysis of the distribution of the n versions of the output over time to determine a drift or an amount of drift. In a particular embodiment, the drift indicates that an actual performance of the ML model is different from an expected performance of the ML model. Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and/or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and/or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and/or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware. In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and/or by end users and a wireless network generally. EXAMPLE EMBODIMENTS Group A Example Embodiments Example Embodiment A1. A method by a user equipment for monitoring a ML model, the method comprising: any of the user equipment steps, features, or functions described above, either alone or in combination with other steps, features, or functions described above. Example Embodiment A2. The method of the previous embodiment, further comprising one or more additional user equipment steps, features or functions described above. Example Embodiment A3. The method of any of the previous embodiments, further comprising: providing user data; and forwarding the user data to a host computer via the transmission to the network node. Group B Example Embodiments Example Embodiment B1. A method performed by a network node for monitoring a ML model, the method comprising: any of the network node steps, features, or functions described above, either alone or in combination with other steps, features, or functions described above. Example Embodiment B2. The method of the previous embodiment, further comprising one or more additional network node steps, features or functions described above. Example Embodiment B3. The method of any of the previous embodiments, further comprising: obtaining user data; and forwarding the user data to a host or a user equipment. Group C Example Embodiments Example Embodiment C1. A method by a user equipment (UE) for monitoring a ML model, the method comprising: for an input sample at a point in time, t, obtaining a number, n, of versions of an output of the ML model; measuring a distribution of the number, n, of versions of the output over time; and based on the distribution, obtaining at least one model monitoring metric. Example Embodiment C2. The method of Example Embodiment C1, wherein the n is greater than or equal to 2, and wherein obtaining the at least one model monitoring metric comprises: for a length, T, time window, determining a distance between two of the n versions of the output. Example Embodiment C3. The method of Example Embodiment C2, comprising: comparing the distance between the two of the n versions of the output to a threshold. Example Embodiment C4. The method of Example Embodiment C2, comprising: comparing the distance between the two of the n versions of the output to a reference distance associated with a reference set. Example Embodiment C5. The method of Example Embodiment C4, comprising: determining model drift based on a difference between the distance between the two of the n versions of the output and the reference distance. Example Embodiment C6. The method of any one of Example Embodiments C4 to C5, wherein the reference set is based on historical data obtained when the ML model was known to be operating properly. Example Embodiment C7. The method of any one of Example Embodiments C2 to C6, wherein T is determined based on a type of the at least one model monitoring metric. Example Embodiment C8. The method of any one of Example Embodiments C2 to C7, wherein obtaining the at least one model monitoring metric comprises using a statistical method to determine the at least one model monitoring metric, wherein the statistical method comprises at least one of: a difference of mean value; a difference at a given CDF, a Kolmogorov statistic method; and a population stability index. Example Embodiment C9. The method of any one of Example Embodiments C1 to C8, comprising performing a time series analysis of the distribution of the n versions of the output over time to determine a drift or an amount of drift. Example Embodiment C10. The method of Example Embodiment C9, wherein the drift indicates that an actual performance of the ML model is different from an expected performance of the ML model. Example Embodiment C11. The method of any one of Example Embodiments C1 to C10, wherein the n versions of the output of the ML model is associated with or used for at least one of: a UE position, beam management, mobility management, and link adaptation. Example Embodiment C12. The method of any one of Example Embodiments C1 to C11, wherein obtaining the n versions of an output of the ML model comprises performing model inference n times. Example Embodiment C13. The method of Example Embodiment C12, wherein performing the model inference n times comprises: obtaining n versions of the input sample, and using a different version of the input sample each time the model inference is performed. Example Embodiment C14. The method of Example Embodiment C13, wherein the input sample comprises a set of elements, and wherein obtaining the n versions of the input samples comprises performing data augmentation on the set of elements. Example Embodiment C15. The method of Example Embodiment C14, wherein performing data augmentation comprises at least one of: dropping a subset of a set of elements in the input sample; setting a subset of the set of elements in the input sample to random noise values; mixing the input sample with previous input data; adding noise to the input sample; applying a random time or frequency shift to elements in the input sample; and changing a resolution of the input sample. Example Embodiment C16. The method of any one of Example Embodiment C12 to C15, wherein performing the model inference n times comprises: obtaining n versions of the ML model, and using a different version of the ML model each time the model inference is performed. Example Embodiment C17. The method of Example Embodiment C16, wherein obtaining the n versions of the ML model comprises using and/or performing model regularization to generate the n versions of the ML model. Example Embodiment C18. The method of Example Embodiment C17, wherein using and/or performing model regularization comprises at least one of: dropping one or more layers; setting at least one model parameter to zero; using a different activation function on at least one layer; adding noise to the input sample; and using and/or setting different hyperparameters. Example Embodiment C19. The method of Example Embodiments C1 to C18, further comprising: providing user data; and forwarding the user data to a host via the transmission to the network node. Example Embodiment C20. A user equipment comprising processing circuitry configured to perform any of the methods of Example Embodiments C1 to C19. Example Embodiment C21. A user equipment configured to perform any of the methods of Example Embodiments C1 to CX. Example Embodiment C22. A wireless device comprising processing circuitry configured to perform any of the methods of Example Embodiments C1 to C19. Example Embodiment C23. A computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments C1 to C19. Example Embodiment C24. A computer program product comprising computer program, the computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments C1 to C19. Example Embodiment C25. A non-transitory computer readable medium storing instructions which when executed by a computer perform any of the methods of Example Embodiments C1 to C19. Group D Example Embodiments Example Embodiment D1. A method by a network node for monitoring a ML model, the method comprising: for an input sample at a point in time, t, obtaining a number, n, of versions of an output of the ML model; measuring a distribution of the number, n, of versions of the output over time; and based on the distribution, obtaining at least one model monitoring metric. Example Embodiment D2. The method of Example Embodiment D1, wherein the n is greater than or equal to 2, and wherein obtaining the at least one model monitoring metric comprises: for a length, T, time window, determining a distance between two of the n versions of the output. Example Embodiment D3. The method of Example Embodiment D2, comprising: comparing the distance between the two of the n versions of the output to a threshold. Example Embodiment D4. The method of Example Embodiment D2, comprising: comparing the distance between the two of the n versions of the output to a reference distance associated with a reference set. Example Embodiment D5. The method of Example Embodiment D4, comprising: determining model drift based on a difference between the distance between the two of the n versions of the output and the reference distance. Example Embodiment D6. The method of any one of Example Embodiments D4 to D5, wherein the reference set is based on historical data obtained when the ML model was known to be operating properly. Example Embodiment D7. The method of any one of Example Embodiments D2 to D6, wherein T is determined based on a type of the at least one model monitoring metric. Example Embodiment D8. The method of any one of Example Embodiments D2 to D7, wherein obtaining the at least one model monitoring metric comprises using a statistical method to determine the at least one model monitoring metric, wherein the statistical method comprises at least one of: a difference of mean value; a difference at a given CDF, a Kolmogorov statistic method; and a population stability index. Example Embodiment D9. The method of any one of Example Embodiments D1 to D8, comprising performing a time series analysis of the distribution of the n versions of the output over time to determine a drift or an amount of drift. Example Embodiment D10. The method of Example Embodiment D9, wherein the drift indicates that an actual performance of the ML model is different from an expected performance of the ML model. Example Embodiment D11. The method of any one of Example Embodiments D1 to D10, wherein the n versions of the output of the ML model is associated with or used for at least one of: a UE position, beam management, mobility management, and link adaptation. Example Embodiment D12. The method of any one of Example Embodiments D1 to D11, wherein obtaining the n versions of an output of the ML model comprises performing model inference n times. Example Embodiment D13. The method of Example Embodiment D12, wherein performing the model inference n times comprises: obtaining n versions of the input sample, and using a different version of the input sample each time the model inference is performed. Example Embodiment D14. The method of Example Embodiment D13, wherein the input sample comprises a set of elements, and wherein obtaining the n versions of the input samples comprises performing data augmentation on the set of elements. Example Embodiment D15. The method of Example Embodiment D14, wherein performing data augmentation comprises at least one of: dropping a subset of a set of elements in the input sample; setting a subset of the set of elements in the input sample to random noise values; mixing the input sample with previous input data; adding noise to the input sample; applying a random time or frequency shift to elements in the input sample; and changing a resolution of the input sample. Example Embodiment D16. The method of any one of Example Embodiment D12 to D15, wherein performing the model inference n times comprises: obtaining n versions of the ML model, and using a different version of the ML model each time the model inference is performed. Example Embodiment D17. The method of Example Embodiment D16, wherein obtaining the n versions of the ML model comprises using and/or performing model regularization to generate the n versions of the ML model. Example Embodiment D18. The method of Example Embodiment D17, wherein using and/or performing model regularization comprises at least one of: dropping one or more layers; setting at least one model parameter to zero; using a different activation function on at least one layer; adding noise to the input sample; and using and/or setting different hyperparameters. Example Embodiment D23. The method of any one of Example Embodiments D1 to D22, wherein the network node comprises a gNodeB (gNB). Example Embodiment D24. The method of any of the previous Example Embodiments, further comprising: obtaining user data; and forwarding the user data to a host or a user equipment. Example Embodiment D25. A network node comprising processing circuitry configured to perform any of the methods of Example Embodiments D1 to D24. Example Embodiment D26. A network node configured to perform any of the methods of Example Embodiments D1 to D24. Example Embodiment D27. A computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments D1 to D24. Example Embodiment D28. A computer program product comprising computer program, the computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments D1 to D24. Example Embodiment D29. A non-transitory computer readable medium storing instructions which when executed by a computer perform any of the methods of Example Embodiments D1 to D24. Group E Example Embodiments Example Embodiment E1. A user equipment for monitoring a ML model, the UE comprising: processing circuitry configured to perform any of the steps of any of the Group A and C Example Embodiments; and power supply circuitry configured to supply power to the processing circuitry. Example Embodiment E2. A network node for monitoring a ML model, the network node comprising: processing circuitry configured to perform any of the steps of any of the Group B and D Example Embodiments; power supply circuitry configured to supply power to the processing circuitry. Example Embodiment E3. A user equipment (UE) for monitoring a ML model, the UE comprising: an antenna configured to send and receive wireless signals; radio front-end circuitry connected to the antenna and to processing circuitry, and configured to condition signals communicated between the antenna and the processing circuitry; the processing circuitry being configured to perform any of the steps of any of the Group A and C Example Embodiments; an input interface connected to the processing circuitry and configured to allow input of information into the UE to be processed by the processing circuitry; an output interface connected to the processing circuitry and configured to output information from the UE that has been processed by the processing circuitry; and a battery connected to the processing circuitry and configured to supply power to the UE. Example Embodiment E4. A host configured to operate in a communication system to provide an OTT service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a cellular network for transmission to a UE, wherein the UE comprises a communication interface and processing circuitry, the communication interface and processing circuitry of the UE being configured to perform any of the steps of any of the Group A and C Example Embodiments to receive the user data from the host. Example Embodiment E5. The host of the previous Example Embodiment, wherein the cellular network further includes a network node configured to communicate with the UE to transmit the user data to the UE from the host. Example Embodiment E6. The host of the previous 2 Example Embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application. Example Embodiment E7. A method implemented by a host operating in a communication system that further includes a network node and a UE, the method comprising: providing user data for the UE; and initiating a transmission carrying the user data to the UE via a cellular network comprising the network node, wherein the UE performs any of the operations of any of the Group A embodiments to receive the user data from the host. Example Emboidment E8. The method of the previous Example Embodiment, further comprising: at the host, executing a host application associated with a client application executing on the UE to receive the user data from the UE. Example Embodiment E9. The method of the previous Example Embodiment, further comprising: at the host, transmitting input data to the client application executing on the UE, the input data being provided by executing the host application, wherein the user data is provided by the client application in response to the input data from the host application. Example Emboidment E10. A host configured to operate in a communication system to provide an OTT service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a cellular network for transmission to a UE, wherein the UE comprises a communication interface and processing circuitry, the communication interface and processing circuitry of the UE being configured to perform any of the steps of any of the Group A and C Example Embodiments to transmit the user data to the host. Example Emboidment E11. The host of the previous Example Embodiment, wherein the cellular network further includes a network node configured to communicate with the UE to transmit the user data from the UE to the host. Example Embodiment E12. The host of the previous 2 Example Embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application. Example Embodiment E13. A method implemented by a host configured to operate in a communication system that further includes a network node and a UE, the method comprising: at the host, receiving user data transmitted to the host via the network node by the UE, wherein the UE performs any of the steps of any of the Group A and C Example Embodiments to transmit the user data to the host. Example Embodiment E14. The method of the previous Example Embodiment, further comprising: at the host, executing a host application associated with a client application executing on the UE to receive the user data from the UE. Example Embodiment E15. The method of the previous Example Embodiment, further comprising: at the host, transmitting input data to the client application executing on the UE, the input data being provided by executing the host application, wherein the user data is provided by the client application in response to the input data from the host application. Example Embodiment E16. A host configured to operate in a communication system to provide an OTT service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a network node in a cellular network for transmission to a UE, the network node having a communication interface and processing circuitry, the processing circuitry of the network node configured to perform any of the operations of any of the Group B and D Example Embodiments to transmit the user data from the host to the UE. Example Embodiment E17. The host of the previous Example Embodiment, wherein: the processing circuitry of the host is configured to execute a host application that provides the user data; and the UE comprises processing circuitry configured to execute a client application associated with the host application to receive the transmission of user data from the host. Example Embodiment E18. A method implemented in a host configured to operate in a communication system that further includes a network node and a UE, the method comprising: providing user data for the UE; and initiating a transmission carrying the user data to the UE via a cellular network comprising the network node, wherein the network node performs any of the operations of any of the Group B and D Example Embodiments to transmit the user data from the host to the UE. Example Embodiment E19. The method of the previous Example Embodiment, further comprising, at the network node, transmitting the user data provided by the host for the UE. Example Emboidment E20. The method of any of the previous 2 Example Embodiments, wherein the user data is provided at the host by executing a host application that interacts with a client application executing on the UE, the client application being associated with the host application. Example Embodiment E21. A communication system configured to provide an over-the- top service, the communication system comprising: a host comprising: processing circuitry configured to provide user data for a UE, the user data being associated with the over-the-top service; and a network interface configured to initiate transmission of the user data toward a cellular network node for transmission to the UE, the network node having a communication interface and processing circuitry, the processing circuitry of the network node configured to perform any of the operations of any of the Group B and D Example Embodiments to transmit the user data from the host to the UE. Example Embodiment E22. The communication system of the previous Example Embodiment, further comprising: the network node; and/or the user equipment. Example Embodiment E23. A host configured to operate in a communication system to provide an OTT service, the host comprising: processing circuitry configured to initiate receipt of user data; and a network interface configured to receive the user data from a network node in a cellular network, the network node having a communication interface and processing circuitry, the processing circuitry of the network node configured to perform any of the operations of any of the Group B and D Example Embodiments to receive the user data from a UE for the host. Example Embodiment E24. The host of the previous 2 Example Embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application. Example Embodiment E25. The host of the any of the previous 2 Example Embodiments, wherein the initiating receipt of the user data comprises requesting the user data. Example Embodiment E26. A method implemented by a host configured to operate in a communication system that further includes a network node and a UE, the method comprising: at the host, initiating receipt of user data from the UE, the user data originating from a transmission which the network node has received from the UE, wherein the network node performs any of the steps of any of the Group B and D Example Embodiments to receive the user data from the UE for the host. Example Embodiment E27. The method of the previous Example Embodiment, further comprising at the network node, transmitting the received user data to the host.

Claims

CLAIMS 1. A method by radio node for monitoring a machine learning, ML, model, the method comprising: for an input sample at a point in time, t, obtaining a number, n, of versions of an output of the ML model; measuring a distribution of the disagreement between the number, n, of versions of the output over time; and based on the distribution of the disagreement between the number, n, of versions of the output over time, obtaining at least one model monitoring metric.

2. The method of Claim 1, wherein obtaining the n versions of an output of the ML model comprises performing model inference n times.

3. The method of Claim 2, wherein performing the model inference n times comprises: obtaining n versions of the input sample; and using a different version of the input sample each time the model inference is performed.

4. The method of Claim 3, wherein the input sample comprises a set of elements, and wherein obtaining the n versions of the input sample comprises performing data augmentation on the set of elements.

5. The method of Claim 4, wherein performing data augmentation comprises performing at least one of: dropping a subset of a set of elements in the input sample; setting a subset of the set of elements in the input sample to random noise values; mixing the input sample with previous input data; adding noise to the input sample; applying a random time or frequency shift to elements in the input sample; and changing a resolution of the input sample.

6. The method of any one of Claim 2 to 5, wherein performing the model inference n times comprises: obtaining n versions of the ML model; and using a different version of the ML model each time the model inference is performed.

7. The method of Claim 6, wherein obtaining the n versions of the ML model comprises using and/or performing model regularization to generate the n versions of the ML model.

8. The method of Claim 7, wherein using and/or performing model regularization comprises at least one of: dropping one or more layers; setting at least one model parameter to zero; using a different activation function on at least one layer; adding noise to the input sample; and using and/or setting different hyperparameters.

9. The method of any one of Claims 1 to 8, wherein: n is greater than or equal to 2, and obtaining the at least one model monitoring metric comprises, for a length, T, time window, determining a magnitude of disagreement between two of the n versions of the output.

10. The method of Claim 9, comprising: comparing the magnitude of disagreement between the two of the n versions of the output to a threshold.

11. The method of Claim 9, comprising: comparing the magnitude of disagreement between the two of the n versions of the output to reference magnitudes of disagreement associated with a reference set.

12. The method of Claim 11, comprising: determining model drift based on a difference between the magnitude of disagreement between the two of the n versions of the output and the reference magnitudes of disagreement.

13. The method of any one of Claims 11 to 12, wherein the reference set is based on historical data obtained when the ML model was known to be operating properly.

14. The method of any one of Claims 9 to 13, wherein the magnitude of disagreement comprises a distance between two of the n versions of the output.

15. The method of any one of Claims 9 to 14, wherein T is determined based on a type of the at least one model monitoring metric.

16. The method of any one of Claims 1 to 15, wherein obtaining the at least one model monitoring metric comprises using a statistical method to determine the at least one model monitoring metric, wherein the statistical method comprises at least one of: a difference of mean value; a difference at a given Cumulative Distribution Function, CDF; a Kolmogorov statistic method; and a population stability index.

17. The method of any one of Claims 1 to 16, comprising performing a time series analysis of the distribution of the n versions of the output over time to determine a drift or an amount of drift.

18. The method of Claim 17, wherein the drift indicates that an actual performance of the ML model is different from an expected performance of the ML model.

19. The method of any one of Claims 1 to 18, wherein the radio node comprises a UE.

20. The method of any one of Claims 1 to 18, wherein the radio node comprises a network node.

21. A radio node for monitoring a machine learning, ML, model, the radio node configured to: for an input sample at a point in time, t, obtain a number, n, of versions of an output of the ML model; measure a distribution of the disagreement between the number, n, of versions of the output over time; and based on the distribution of the disagreement between the number, n, of versions of the output over time, obtain at least one model monitoring metric.

22. The radio node of Claim 21, wherein when obtaining the n versions of an output of the ML model the radio node is configured to perform model inference n times.

23. The radio node of Claim 22, wherein when performing the model inference n times the radio node is configured to: obtain n versions of the input sample; and use a different version of the input sample each time the model inference is performed.

24. The radio node of Claim 23, wherein the input sample comprises a set of elements, and wherein when obtaining the n versions the radio node is configured to perform data augmentation on the set of elements.

25. The radio node of Claim 24, wherein when performing data augmentation the radio node is configured to perform at least one of: dropping a subset of a set of elements in the input sample; setting a subset of the set of elements in the input sample to random noise values; mixing the input sample with previous input data; adding noise to the input sample; applying a random time or frequency shift to elements in the input sample; and changing a resolution of the input sample.

26. The radio node of any one of Claim 22 to 25, wherein when performing the model inference n times the radio node is configured to: obtain n versions of the ML model; and use a different version of the ML model each time the model inference is performed.

27. The radio node of Claim 26, wherein when obtaining the n versions of the ML model the radio node is configured to use and/or perform model regularization to generate the n versions of the ML model.

28. The radio node of Claim 27, wherein when using and/or performing model regularization the radio node is configured to perform at least one of: dropping one or more layers; setting at least one model parameter to zero; using a different activation function on at least one layer; adding noise to the input sample; and using and/or setting different hyperparameters.

29. The radio node of any one of Claims 21 to 28, wherein: n is greater than or equal to 2, and when obtaining the at least one model monitoring metric the radio node is configured to determine, for a length, T, time window, a magnitude of disagreement between two of the n versions of the output.

30. The radio node of Claim 29, wherein the radio node is configured to compare the magnitude of disagreement between the two of the n versions of the output to a threshold.

31. The radio node of Claim 29, wherein the radio node is configured to compare the magnitude of disagreement between the two of the n versions of the output to reference magnitudes of disagreement associated with a reference set.

32. The radio node of Claim 31, wherein the radio node is configured to determine model drift based on a difference between the magnitude of disagreement between the two of the n versions of the output and the reference magnitudes of disagreement.

33. The radio node of any one of Claims 31 to 32, wherein the reference set is based on historical data obtained when the ML model was known to be operating properly.

34. The radio node of any one of Claims 29 to 33, wherein the magnitude of disagreement comprises a distance between two of the n versions of the output.

35. The radio node of any one of Claims 29 to 34, wherein T is determined based on a type of the at least one model monitoring metric.

36. The radio node of any one of Claims 29 to 35, wherein obtaining the at least one model monitoring metric comprises using a statistical method to determine the at least one model monitoring metric, wherein the statistical method comprises at least one of: a difference of mean value; a difference at a given Cumulative Distribution Function, CDF; a Kolmogorov statistic method; and a population stability index.

37. The radio node of any one of Claims 21 to 36, wherein the radio node is configured to perform a time series analysis of the distribution of the n versions of the output over time to determine a drift or an amount of drift.

38. The radio node of Claim 37, wherein the drift indicates that an actual performance of the ML model is different from an expected performance of the ML model.

39. The radio node of any one of Claims 21 to 38, wherein the radio node comprises a UE.

40. The radio node of any one of Claims 21 to 38, wherein the radio node comprises a network node.