EP4619900A1 - Method and system for data labeling in a cloud system through machine learning - Google Patents

Abstract

Embodiments include methods, electronic device, storage medium, and computer program to label of data points of a cloud system automatically. In one embodiment, a method comprises: obtaining data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning, using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

Description

SPECIFICATION

METHOD AND SYSTEM FOR DATA LABELING IN A CLOUD SYSTEM THROUGH MACHINE LEARNING

TECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of cloud computing, and more specifically, to data labeling in a cloud system through machine learning.

BACKGROUND ART

[0002] Recent advancements in machine learning have brought about transformative applications to the computational problem-solving landscape. However, supervised machine learning solutions require labeled data for training. Though unsupervised machine learning solutions do not require labeled data for training, they still need it for evaluation and their applicability and performance is often inferior to those of supervised learning models. While acquiring labeled data is necessary, data labeling is a significant bottleneck in machine learning. [0003] Data labeling entails the assignment of one or more informative labels to the data points, which provide the context of learning to a machine learning algorithm. Manual data labeling is expensive, error-prone, and susceptible to human biases. Various solutions across the literature have organized the labeling experience as a human-machine collaboration to improve label quality and reduce manual labeling efforts. However, manual labeling is challenging in some domains, even for domain experts, and hence unfeasible to perform in scale.

[0004] One such problematic domain is fault detection in cloud computing environments. Faults are inevitable in any cloud environment, and if not detected and mitigated in time, they can accumulate and propagate, leading to severe performance degradation and downtime. Of the several techniques for fault detection and prediction in cloud environments, machine learningbased solutions have lately received the utmost attention. Meticulously designed machine learning solutions can identify and learn the correlations between various metrics obtained from the cloud system and corresponding faults. The application and evaluation of these machine learning solutions require a dataset of cloud system metrics annotated with the corresponding system state, which can be a normal system state or a system state with one or more faults.

[0005] While manual labeling can be the most accurate method of labeling data points in most scenarios, manual annotation of cloud system metrics with the corresponding faults is a challenging task, even for domain experts. Efficient data labeling solutions are thus desirable for fault detection in cloud environments. SUMMARY OF THE INVENTION

[0006] Embodiments include methods, electronic device, storage medium, and computer program to label data points of a cloud system automatically. In one embodiment, a method comprises: obtaining data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning, using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

[0007] In one embodiment, an electronic device comprises a processor and machine-readable storage medium that provides instructions that, when executed by the processor, are capable of causing the processor to perform: obtaining data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning, using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

[0008] In one embodiment, a machine-readable storage medium that provides instructions that, when executed by a processor, are capable of causing the processor to perform: obtaining data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning, using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

[0009] Embodiments of the invention enable automated labeling of cloud system metrics, which is a challenging task to do manually, even for domain experts. They allow discovery of new patterns in the cloud system metrics that manifest into faults and obtain the corresponding labeled cloud system metrics, the discovery through informativeness measures is not feasible with prior fault injection mechanisms. Furthermore, the labeling may respond to concept drift and obtain corresponding labels through test suite running test cases and feed the labeling results to retrain the classifiers. Through running test cases to detect fault states of a cloud system and feeding the resulting labels to a classifier, the classifier may be trained to label all the detectable fault states. Such automated data labeling greatly reduces cloud resource consumption (since test suites are invoked only when needed) and human efforts required to label data points.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0011] Figure 1 illustrates a framework for automated labeling of data points of a cloud system per some embodiments.

[0012] Figure 2 illustrates a management cloud system for which labels are applied to data points per some embodiments.

[0013] Figure 3 illustrates training of a classifier deep neural network per some embodiments. [0014] Figure 4 illustrates an exemplary implementation of the automated labeling framework per some embodiments.

[0015] Figure 5 is a flow diagram illustrating the operations to label data points of a cloud system automatically per some embodiments.

[0016] Figure 6 illustrates an electronic device implementing adaptive fault remediation per some embodiments.

[0017] Figure 7 illustrates an example of a communication system per some embodiments.

[0018] Figure 8 illustrates a user equipment (UE) per some embodiments.

[0019] Figure 9 illustrates a network node per some embodiments.

[0020] Figure 10 is a block diagram of a host, which may be an embodiment of the host of Figure 7, per various aspects described herein.

[0021] Figure 11 is a block diagram illustrating a virtualization environment in which functions implemented by some embodiments may be virtualized.

[0022] Figure 12 illustrates a communication diagram of a host communicating via a network node with a user equipment (UE) over a partially wireless connection per some embodiments.

DETAILED DESCRIPTION

[0023] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and/or is implied from the context in which it is used. All references to a/an/the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and/or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features, and advantages of the enclosed embodiments will be apparent from the following description.

Overview of Data Labeling Techniques and Fault Injection

[0024] Several types of data labeling techniques have been introduced over the years, and they are reviewed briefly herein below.

Self-Labeled Techniques

[0025] One technique is to exploit the existing (usually a small set of) labeled data to infer the labels for the remaining (a much larger set of) unlabeled data. This category of semi-supervised learning techniques, called self-labeled techniques, aims at generating labels for the unlabeled data by trusting one's own predictions. For example, self-labeled techniques may follow the principle of inductive learning to discover a general hypothesis from the labeled data points that can be applied to any newly obtained unlabeled data points to generate their labels.

[0026] The fundamental ideas of self-labeled techniques have been extended in several ways. For example, systems and methods are introduced for automatically labeling features in unlabeled video frames in a video sequence based on features in labeled video frames in the same video sequence, given that the labeled video frame is temporally close to the unlabeled video frame.

[0027] Additionally, images can be automatically labeled or segmented using self-labeled techniques. Training images and labels are assigned into a plurality of clusters, and representative templates are derived for each cluster composed of a representative image and its representative label. A model is then trained over training images and labels that produces suboptimal labels for the unlabeled images, and optimal labels for the unlabeled images are generated using the derived suboptimal labels and representative templates of the clusters.

[0028] Furthermore, curriculum learning models may be used for data labeling. In that case, pseudo-labels are assigned to unlabeled data points based on a similarity measure in an embedding space to labeled data points. A curriculum learning model is then trained using the pseudo-labeled data points and further fine-tuned over the labeled and task-specific data points sequentially to enhance the model's classification or labeling accuracy.

Graph-Based Label Propagation Techniques

[0029] Graph-based label propagation techniques aim at deriving a weighted graph structure among the data points based on their similarities. Then these techniques exploit this graph structure to propagate labels from the labeled data points to the unlabeled data points. Graphbased label propagation techniques follow the principle of transductive learning to exploit the similarities between labeled and unlabeled data points to deduce the labels for these unlabeled data points. Transductive learning techniques do not generate a general hypothesis and labeling any newly obtained unlabeled data points will require the entire algorithm to be executed again.

Active Learning

[0030] Active learning, a crowd-based technique, focuses on selecting the most interesting data points to be delegated to the crowd for labeling to reduce the manual labeling effort. It usually involves one or more machine learning models to make label predictions for the unlabeled data points. The key challenge here is the selection of data points whose labels will be most informative to the underlying machine learning algorithm and should be delegated to the human labelers for labeling. Here, the human labelers are expected to be very accurate. The most straightforward approach is uncertainty sampling, which chooses the next unlabeled data point with the highest model prediction uncertainty.

Crowdsourcing

[0031] Another crowd-based technique is crowdsourcing, which, in contrast to active learning, leans toward employing many workers who are not necessarily experts for manual labeling. Such a technique tries to improve the reliability of the labeling process, acknowledging that the workers may make mistakes. Crowdsourcing techniques are most befitting to the labeling tasks where general human wisdom produces accurate results, for instance, image annotation and text categorization.

Weak Supervision

[0032] Another approach to data labeling is weak supervision, where the idea is to generate large quantities of data labels, which are not necessarily as accurate as manually assigned labels. Nevertheless, the large amount of labeled data generated suffices for the inaccuracies within to train a model with reasonable final accuracy. One form of weak supervision is data programming, as in the Snorkel system, wherein multiple programmable labeling functions are defined. These labeling functions are combined into generative models or majority voting mechanisms to generate weak labels.

[0033] While data labeling techniques such as the ones discussed above are numerous, they are not inherently befitting for automated data labeling for fault detection in cloud environments. For example, self-labeled and graph-based label propagation techniques can be used to generate fault data labels for the unlabeled data points following similar data distribution as known faults. Weakly supervised learning has comparatively inferior generalization and accuracy. However, the labeled dataset is often obtained through fault injection, and faults identified with such techniques are restricted to those that are already known. Solutions that rely on human experts, such as those based on crowd-based solutions, are suboptimal since such approach is not only slow, expensive, and error-prone, but also challenging to correctly label the cloud system metrics with the corresponding faults manually.

Fault Injection

[0034] Most machine learning-based solutions for fault detection in cloud environments acquire the labeled cloud system metrics data by fault injection mechanism. The fault injection mechanism of labeling entails manually injecting faults into the system and assigning the cloud system metrics collected in the fault injection time frame with the injected faults as labels.

[0035] Yet the fault injection mechanism has several issues. It is undesirable to purposefully inject a fault to a production system, which means that only controlled and limited faults can be injected. This leads to fault detection models trained on such datasets being limited to detecting only these artificial faults and hence have limited applicability to faults in real-world scenarios. Additionally, due to the length of the fault injection/recovery cycle, generating the labeled data takes quite a long time. Furthermore, such solutions have limited applicability in real systems due to data drift - a change in the statistical nature of the data severely affects the accuracy of the model, requiring retraining with new data. Finally, the manual injection is not scalable and error prone.

Automated Labeling Framework

[0036] Embodiments of the invention provide a framework for automated labeling of data points collected from a cloud system (e.g., a cloud platform offered by a cloud service provider) that measure metrics to track operational status of the cloud system. The labels indicate whether the system is in a normal system state or in a state with one or more faults. The framework automatically annotates these data points with the appropriate system state labels. While examples herein are related to data labeling for fault detection in cloud environments, embodiments of the invention have applications beyond fault detection in fields like performance management, network security, and many more. For example, the framework may be extensible to any system that exposes metrics representing the system’s state and used for indicating whether there is an ongoing security threat or whether applications are not meeting their key performance indicator (KPI) targets.

[0037] Figure 1 illustrates a framework for automated labeling of data points of a cloud system per some embodiments. Framework 100 includes a managed cloud system 102, which is monitored by a cloud monitoring system 104. The cloud monitoring system 104 collects data points from the managed cloud system 102, where each data point includes measurements of one or more metrics to track operational status of the managed cloud system 102, and the data points are also referred to as metric data points or metrics. The data points include ones collected as the managed cloud system 102 is operating, and as they are generated in real time based on the operational status of the managed cloud system 102, they form live metric streams, e.g., each feature to track the operational status of the managed cloud system 102 (e.g., consumption of executing resources such as central processing unit (CPU) or graphics processing unit (GPU) usage or temperature of a server on the cloud system) forming one live metric stream. The data points may also include historical data points (referred to as historical metrics) that have been collected earlier from the managed cloud system 102.

[0038] The data points collected in the cloud monitoring system 104, both historical data points and live metric streams, are preprocessed at data preprocessing module 106. These data points are provided to an information content based active learning logic 115. The information content based active learning logic 115 may be implemented in an electronic device, such as electronic device 602, discussed herein below in detail.

[0039] As shown, in some embodiments, the live metric streams are fed into an active learning classifier 108, which implements one or more machine learning models. The active learning classifier 108 classifies the data points with the live metric streams into different classes of states, e.g., the normal state, an unknown state, a known type of fault state (e.g., a first type of fault of virtual machine exhaustion, second type of fault of excessive packet delay) and probabilities of a data point being in a given state (shown as class probabilities).

[0040] The classification and probabilities are provided to a measures of informativeness module 112. An informativeness measure indicates the ability of a data point to reduce the generalization error of classification by the active learning classifier 108 and increase the confidence level of the classification of the data point.

[0041] The class probabilities and measures of informativeness of the data points are then provided to a label assignment module 116, which assigns labels to the data points and the labeled data points are the output from the label assignment module 116. When the label assignment module 116 is unsure about a data point, it consults a test suite module 118, which determines the label applicable to the data point.

[0042] The test suite module 118 includes test cases to be ran on the managed cloud system 102, and the test cases, once ran on the managed cloud system 102, will determine the operational status of the managed cloud system 102, and based on the determination, can give proper label to a data point provided by the label assignment module 116 as either the normal state or a known type of fault state. Since the test suite, after running the test cases on managed cloud system 102, may arbitrate the labels applied to data points that the label assignment module 116 is unsure about, and/or determine the labels to be applied to data points when the label assignment module 116 cannot, the test suite module 118 is also referred to as the active learning oracle that provides the knowledge of the true operational status of the managed cloud system 102.

[0043] Running the test cases in the test suite 118 consumes resources on the managed cloud system 102, thus it negatively impacts the performance of the managed cloud system 102. By incorporating the information content based active learning logic 115 to provide classification first, embodiments of the invention may significantly reduce the involvement of the test suite 118, which is now triggered only when the label assignment 116 cannot label data points to a fault state with sufficient confidence and/or have to label a data point corresponding to an unknown state.

[0044] The labeled data points (with or without the help of test suite 118) may be saved in a data points buffer 120, which may then be used to train or fine-tune the active learning classifier at the training module 110. The training module 110 takes the historical metrics from the data preprocessing module 106 and the labeled data points from the data points buffer 120 to train the active learning classifier, and once it’s trained, fine-tune it to make the classification by the active learning classifier 108 more accurate. The training model may also be used to set one or more thresholds for measures of informativeness at 114, which provides/updates the thresholds to the measures of informativeness module 112.

[0045] The labeled data points from the label assignment module 116 may be used to enhance performance of the managed cloud system 102 in various ways. For example, the labeled data points can be used to train and improve machine learning models for fault detection and prediction in cloud environments, facilitating prompt identification and alleviation of faults to ensure high availability and compliance with service level agreements (SLAs) for clients of the managed cloud system 102.

[0046] Note that while the information content based active learning logic 115 as shown includes four modules at references 108 to 114, some embodiments may have the information content based active learning logic 115 to be integrated with one or more other modules to be a data point labeling system 150, which additionally include one or more systems/entities such as the cloud monitoring system 104, the data preprocessing module 106, the label assignment module 116, test suite 118, and data points buffer 120. Furthermore, the information content based active learning logic 115 may be integrated with the cloud monitoring system 104 to provide labels to the collected data points, and the electronic device implementing the integrated system may store the labels along with the collected data points, which makes the collected data points easier to be further processed to examine the performance of the managed cloud system 102. [0047] More details are discussed herein below based on some embodiments regarding the managed cloud system, the cloud monitoring, data preprocessing, and active learning done through the information content based active learning logic.

Managed Cloud System

[0048] The managed cloud system is a cloud platform with a deployed workload consuming the cloud resources with the objective of providing the desired services while meeting their specifications. Figure 2 illustrates a managed cloud system for which labels are applied to data points per some embodiments. In system 200, the managed cloud system 252 is monitored through cloud monitoring, which generates data points that a data point labeling system 250 labels. The data point labeling system 250 may be similar to the data point labeling system 150 but may include a cloud monitoring system such as the cloud monitoring system 104 in some embodiments. Alternatively, the cloud monitoring system may be included in the managed cloud system 252.

[0049] As shown, the managed cloud system 252 includes (1) cloud-native microservices/ services module 202, which interacts with web user requests 204, (2) deployed workload 206 (e.g., online boutique application pods 208), (3) container orchestration platform 210, (e.g., a Kubernetes environment 212, all of which may be fabricated on top of virtual machines 214 (e.g., master which serves as Kubernetes master node, worker- 1 and worker-2 which serve as Kubernetes worker nodes, and admin which serves other cloud administrative purposes, at reference 216) and provisioned in a cloud computing platform 218 (e.g., an OpenStack cloud 220). The microservices/services 202 may generate a real-time workload as the deployed workload 206 on the cloud to serve web user requests 204 (e.g., HTTP user requests) while meeting the acceptable response time specifications. Note the managed cloud system 252 may be implemented along with or in a communication system such as the communication system 700.

[0050] The cloud infrastructure, deployed applications, and other intrinsic components of the managed cloud system, as shown in the figure, generate metrics that track the operational status of the managed cloud system 252, including service uptime, CPU usage, memory usage, disk usage, swap usage, average requests, average response times, and many more that provide insights into the availability, health, and performance of the managed cloud system 252. For instance, Kubernetes environment 212, in the example managed cloud system 252, exposes several thousand metrics for its node components like Kubelet and proxy and its control plane components like application programming interface (API) server, controller manager, and scheduler. These metrics are monitored by a cloud monitoring system (e.g., the cloud monitoring system 104). Cloud Monitoring System

[0051] A managed cloud system may be instrumented with a cloud monitoring system that collects metrics from configured components of the managed cloud system at specified intervals. For instance, the Prometheus monitoring toolkit can be configured to scrape the example managed cloud system every 10 seconds for metrics of Container Advisor (referred to as cAdvisor that collects, aggregates, processes, and exports information about running containers), where the metrics are exposed by Kubelet which expose resource usage and performance data from running containers. For brevity of case in point, we expand on cAdvisor metrics that can reveal CPU and memory faults occurring at node and pod levels as shown in Table 1 and Table 2 below.

Table 1: Examples of Gauge Type cAdvisor Metrics Table 2: Examples of Counter Type cAdvisor Metrics

Data Preprocessing

[0052] A data preprocessing module transforms raw metric data points into a more suitable form for a more performant application of a machine learning algorithm. For the metrics obtained from a cloud monitoring system, some preprocessing steps may be performed. However, the discussion herein does not provide an exhaustive list but only exemplifies some quintessential data preprocessing operations.

[0053] A cloud monitoring system may collect two types of metrics from a cloud monitoring system, gauge and counter. A gauge-type metric (an example shown in Table 1) has a numerical value that can arbitrarily go up and down over time. On the other hand, a counter-type metric (an example shown in Table 2) has a cumulative value that increases monotonically over time or is reset to zero on restart. As a counter-type metric is not suitable for learning in its raw form, data preprocessing may transform it into a gauge-type metric. In some embodiments, this is achieved by replacing the value of a counter-type metric at any timestamp with its difference from its value at the previously recorded timestamp.

[0054] The metrics obtained from a cloud monitoring system usually have values on different scales. We need to bring all the metrics on the same scale so that the machine learning algorithm is not unfairly biased to metrics with generally higher values. In some embodiments, this is achieved by normalizing the metrics to the range [0, 1], One approach to normalization is using MinMaxScaler. The MinMaxScaler transformation for a metric value m of a metric f can be defined as:

(m - fmir) mscaled = — - - — —

(fmax - fmin) [0055] In Formula (1), fmin and fmax are the minimum and maximum values of the metric f respectively.

Active Learning Classifier

[0056] A managed cloud system can produce a multitude of metrics with diverse characteristics and embodied information. The application of traditional machine learning algorithms over these metrics entails domain expertise and efforts to engineer the optimum features for learning. Moreover, the features are often engineered specifically to the task they are designed for and might not transfer well to other tasks. As opposed to traditional machine learning algorithms, deep learning algorithms can self-leam hierarchical feature representations, alleviating the need for domain expert feature engineering. In addition, the learned feature representations are better transferable to new tasks within the same domain, facilitating the ability to transfer learn new emerging faults. Hence, a deep learning classifier is a good fit for the automated labeling framework.

[0057] The cloud monitoring system produces a time series of metrics that reveals the temporal changes in the managed cloud system that manifest into faults. An elementary approach is to classify the metrics at each point in time separately, but it overlooks the information embodied in the temporal sequence of the metrics. As it is of essence that the classifier learns this temporal dynamic behavior to identify the system states precisely, we need to employ a time series classifier. Recurrent neural network variants like long short-term memory (LSTM) and gated recurrent units, and convolutional neural networks (CNN) are some embodiments of a time series classifier. To realize the time series classification, some embodiments apply the moving window technique over the metric time series to obtain an input time series window x of metric data points for the classifier. We denote an element of x by x_tj, where i E {1, 2, ... , L] and j E {1, 2, ... , F] if L is the number of metric data points in the timeseries window and F is the number of metrics in each metric data point.

[0058] At any point in time, the managed cloud system can either be in a normal system state or a system state with one or more faults. The prospect of multiple faults occurring simultaneously necessitates the use of a multi-label classifier to annotate x with multiple fault labels. For illustration, consider an exemplary managed cloud system in a Kubernetes environment as shown in Figure 2. If the memory pressure on a node exceeds an eviction threshold, the Kubelet tries to reclaim node memory resources by evicting application pods. In a scenario where the memory pressure on worker-1 results in the eviction of Frontend and Recommendation Service pods, the classifier should assign Frontend Out of Memory, Recommendation Service Out of Memory, and Worker 1 Memory Overload classes to the corresponding time series windows of metric data points. Detection of multiple concurrent faults is a challenging problem owing to the unavailability of data for such scenarios, which makes it indispensable to capture and label such data points.

[0059] In this embodiment, in congruence with the aforementioned requisites, we employ a time series multi-label deep learning classifier in our active learning settings to return probabilities of x belonging to the classes:

[0060] Unknown System State US, which determines if the metric data point belongs to an unseen and novel metric data distribution.

[0061] Normal System State NS, which corresponds to a normally functioning state of the cloud system.

[0062] Fault State F_k, where k G {1, 2, ... , M} could be any of the M previously encountered faults. For brevity of discussion, we consider F_k to be a CPU or memory fault in one of the pods or nodes in the example managed cloud system, for instance, Frontend CPU Throttling, Recommendation Service Out of Memory, Worker 1 CPU Overload, and so on.

[0063] So, the output of the classifier is Y' = {p{y_us)> P where y_t G {0, 1} is the label indicating if x belongs to class i, p(yt) is the predicted probability of x belonging to class i, and i G {US, NS, F_lt F₂, ... , F_M}.

[0064] For better comprehension, we discuss a classifier architecture that satisfies all the abovementioned requisites and the associated training procedure, although the proposed framework is not limited to this classifier architecture. The illustrative classifier deep neural network comprises:

[0065] Encoder: £■(%) is an LSTM-based neural network for generating a compressed feature representation z of a time series window x of metric data points lying in a high dimensional metric space.

[0066] Decoder: £)(z) is an LSTM neural network to map the compressed feature representation z back to the input time series window x of metric data points.

[0067] Classifier: C(z) is a feed-forward neural network to generate probabilities of the compressed feature representation z belonging to the above-mentioned classes.

[0068] In this embodiment, long short-term memory (LSTM) networks may be used as they enable the modeling of long-term temporal dependencies in time series data by addressing the vanishing gradient problem in traditional recurrent neural networks.

[0069] For example, E(x) may generate compressed feature representation z in a lower dimensional latent space of a time series window x of metric data points lying in a high dimensional metric space. In one embodiment, E(x) can be a stack of LSTM layers, where each layer has fewer outputs than its inputs. D(z) is a deep neural network for mapping the compressed feature representation z in a lower dimensional latent space back to the original time series window x of metric data points lying in a high dimensional metric space. In one embodiment, D(z) can be a stack of LSTM layers, where each layer has more outputs than its inputs. C(z) is a feed-forward neural network for generating probabilities of the compressed feature representation z of the time series window x of metric data points belonging to one or more state classes. The E(x)-D(z) neural network aims to learn a compressed feature representation z that embodies all crucial information in input x to reconstruct the input x appropriately. The E(x)-C(z) neural network aims to learn a compressed feature representation z that incorporates all relevant information in input x to classify the input x accurately.

Classifier Training

[0070] Figure 3 illustrates training of a classifier deep neural network per some embodiments. The training system 300 is split into two parts, which are to train the network between the encoder 304 to decoder 306 and to train network between the encoder 304 to classifier 316, through the input data 302.

Encoder-Decoder Training

[0071] In some embodiments, the encoder 304 generates an encoding z of dimension D for a time series window x of metric data points (represented by compressed feature representation 308), where each value in z belongs to the range (0, 1). The decoder 306 regenerates the input x from the encoding to produce reconstructed input 310. Here, the training aims to minimize the reconstruction error. We define the reconstruction error as binary cross-entropy loss: [0072] In Formula (2), N is the batch size, L is the number of metric data points in the timeseries window, F is the number of metrics in each metric data point, Xj is a metric’s original value, and x'_tj is its reconstructed value.

Encoder-Classifier Training

[0073] Classifier 316 has to address two subproblems. Firstly, it needs to generate probability measures of the feature representation z belonging to the previously encountered system states, which are NS and F_k, and that comprises a multi-label classification problem. Secondly, it needs to generate a probability measure that the feature representation z belongs to US or a previously unencountered metric data distribution, which comprises a one-class classification problem.

[0074] The key challenge in one-class classification is the absence of negative class data points. To address it, we append the features generated z with pseudo-negative data generated from uniform distribution in the same dimensionality D and range (0, 1) as z. To an input batch size of N samples from metric data space, we append N samples from pseudo-negative data space, before feeding them to the classifier. Figure 3 shows a pseudo-negative data from uniform distribution 312 is appended at an appendix module 314 to the compressed data from the encoder 304.

[0075] For a sample from metric data space, the loss propagates from the classifier to the encoder. On the other hand, for a sample from pseudo-negative data space, the loss only propagates to the classifier. Here the training aims to minimize the cross-entropy loss: [0076] In Formula (3), N is the batch size, F + 2 is the total number of output classes, y_t E {0, 1} is the label indicating if the input sample belongs to class i, p(yt) is the predicted probability of the input sample belonging to class i, and i E {US, NS, F_k}. For a sample from the metric data space, the label set looks like Y = {0, y_ws,y_F1,y_F2, ... ,y_FM}, where y_NS and y_Ffe are mutually exclusive. For a sample from the pseudo-negative data space, the label set looks like Y = {1, 0, 0, 0, ... , 0}. Here, the first element in Y belongs to y_us, the second element in Y belongs to y_NS, and the later elements in Y belong to y_Ffe.

[0077] The Encoder-Decoder and Encoder-Classifier networks are trained simultaneously for each batch of input samples in some embodiments. While the Encoder-Classifier network performs the crucial task of giving out class probabilities, Encoder-Decoder network makes sure that the learned feature representation incorporates all characteristics of the input samples to ensure better transferability to emerging fault classes and data distributions down the line.

Active Learning Measures of Informativeness

[0078] We follow the active learning approach of selecting the most informative samples from the metric data space to be labeled by the test suite. While shrewdly defined test cases can identify the faults leading to service specification failures with high accuracy, they tend to be slow and cloud resource expensive. A mechanism is thus established where the test suite is only triggered when the classifier encounters new metric data distributions. Essentially, there are two scenarios where we trigger the test suite, that are the following:

[0079] (1). The uncertainty associated with the predicted probability of any class i, where i E {NS, F_k} is above a pre-configured threshold a_t. In one embodiment, we can quantify uncertainty as the binary entropy: [0080] Here BEi is the uncertainty associated with the predicted probability for class i.

[0081] In one embodiment, we can configure the thresholds c manually based on the confidence desired from the classifier's predictions, for instance, each of them to a binary entropy of 0.469. Here, a binary entropy of less than 0.469 for a data point corresponding to a class implies that the classifier has more than 90% confidence in its prediction.

[0082] In another embodiment, we can configure the threshold c for each class as the average binary entropy for that class over the metric data points TD _t with the label i used for training the classifier:

[0083] (2). The probability of the input sample belonging to the class US is above a preconfigured threshold ?.

[0084] In one embodiment, we can configure the thresholds [J manually based on the confidence desired from the classifier's prediction, for instance, to a probability of 0.1. Here, a probability of less than 0.1 implies that the classifier is more than 90% confident that the data point does not belong to the class US.

[0085] In another embodiment, we can configure the threshold [J as the average predicted probability for the class US over the metric data points TD _US without the label US used for training the classifier:

[0087] In Formula (7), function T(x) determines if test suite is triggered for the input sample x or not.

Test Suite

[0088] The test suite is a set of tests, primarily functional tests, intended to programmatically identify the faults in the managed cloud system in real time. More specifically, in the exemplified use case, these functional tests are intended to identify the services that fail to meet the functional specifications.

[0089] In this embodiment, we consider the Online Boutique application from the example managed cloud system, a web-based e-commerce application composed of various microservices running in pods. Some of the constituent microservices are directly called upon by user actions, whereas some are called upon by other microservices in turn. For convenience, we will call the former user microservices, and the latter helper microservices. We define functional tests to check the compliance of user microservices with the specifications. Specifically, we check that the user microservices are available and have acceptable response times. The failure of a functional test corresponding to a user microservice implies that either the user microservice has failed, or one or more of its helper microservices have failed.

[0090] We define a relationship matrix between functional test cases and microservices to identify the microservices whose failure can lead to the failure of a functional test case. An example of the relationship matrix is shown in Table 3. When a functional test case fails, we can annotate the corresponding metric data points with the associated microservice failures. For instance, when homepage test case fails, we can annotate the corresponding metric data points with the labels Cart Service Failure, Currency Service Failure, Product Catalog Service Failure, Frontend Service Failure, and Shipping Service Failure.

Table 3: Functional Test - Microservice Relationship Matrix

[0091] To further narrow down to the root cause of the service failures, we can define root cause identification tests intended to identify the factors that caused the nonconformance to the specifications. Whenever a functional test case fails, we execute the root cause identification tests to identify the underlying resource faults in the related pods or worker virtual machines that led to this failure. Using root cause identification tests, we can further narrow down and qualify the labels for the corresponding metric data points, such as Frontend Pod Out of Memory, Recommendation Service Pod CPU Throttle, or Worker-1 Memory Overload.

[0092] Alternatively/additionally, some embodiments may integrate test suites like OpenStack Tempest or Kubemetes E2E Tests in the proposed framework.

An Exemplary Implementation of the Automated Labeling Framework

[0093] Figure 4 illustrates an exemplary implementation of the automated labeling framework per some embodiments. The exemplary implementation in system 400 (implementing another data point labeling system) provides further details of the automated labeling framework shown in Figure 1, and the managed cloud system 102, cloud monitoring system 104, and data preprocessing module 106 remain the same. The metrics data points are shown as the metric stream 406, which is fed into a time series multi-label deep learning classifier 408 as an embodiment of the active learning classifier 108.

[0094] The time series multi-label deep learning classifier 408 classifies each metrics data point to one or more of the classes 410, including an unknown system state (US), a normal system state (NS), and one of fault states (Fl, F2,. . FM). The classification to each state may be associated with a probability (p(yi)) at reference 416. Note that the output of a classifier is discussed herein relating to Y' where y_£ G {0, 1} is the label indicating if x belongs to class i, p(yt) is the predicted probability of x belonging to class i, and i G {US, NS, F_lt F₂, ..., F_M}.

[0095] After the classification, the measures of informativeness module 432 identifies the informative metric data points from the metric data stream for delegation to the corresponding test suite 422, which is the active learning oracle for labeling. In some embodiments, the test suite 422 only runs test cases upon the delegated metric data points being received from the time series multi-label deep learning classifier 408. The measures of informativeness module 432 identifies metric data points to be informative when (1) the metric data points correspond to emerging fault patterns in the managed cloud system 102, and/or (2) the metric data points are collected after a concept drift in the managed cloud system 102. Note that concept drift occurs when the relationship between the input variables and the output variables of the deep learning classifier changes over time, leading to a degradation of model performance.

[0096] Concept indicates the joint probability distribution of a machine learning model’s inputs (X) and outputs (Y), and it may be represented as P(X, Y) = P(Y) P(X\Y) = P(X) P(Y\X). Concept shift happens when the joint distribution of inputs and outputs changes: Pti (X )') P^ Y).

[0097] Metrics data points indicating an emerging fault pattern and/or concept drift carry higher information content than metric data points from the data distributions already learned by the time series multi-label deep learning classifier 408. We thus define active learning measures of informativeness derived from the predicted class probabilities of the classifier to quantify the information carried by the metric data points.

[0098] Furthermore, we configure thresholds on these informativeness measures to delegate the metric data points with a higher information content to the test suite 422 for labeling. In one embodiment, we can quantify informativeness by the binary entropies associated with the class probabilities predicted by the classifier. We can further configure the binary entropy threshold for each class as the average binary entropy for that class over the metric data points used for training the classifier. The threshold [J as shown at reference 412 is explained relating to Formula (6), and the threshold c as shown at reference 414 is explained relating to Formula (5). [0099] When at least one of the thresholds is crossed, the test suite 422 is triggered to run test cases on the managed cloud system 102 to determine the operational status of the managed cloud system 102. Based on the determination, the test suite 422 gives proper labels to data points that are deemed to represent a new class (reference 418) and/or that are classified with a high uncertainty (reference 420), and the labels with an acceptable certainty are provided at reference 424. These data points are given the labels at the buffer informative data points 426 otherwise.

[00100] When it is determined at reference 428 that enough data points are labeled by the test suite 422, these labels and corresponding data points are used to update parameters of the time series multi-label deep learning classifier 408, as shown at reference 430. In some embodiments, updating the parameters of the time series multi-label deep learning classifier 408 is through transfer leaning, which configures and updates the parameters of the time series multi-label deep learning classifier 408 based on the newly/adjusted labels by the test suite 422.

Operations per Some Embodiments

[00101] Figure 5 is a flow diagram illustrating the operations to label data points of a cloud system automatically per some embodiments. The operations of method 500 are performed by a data point labeling system, such as the ones at reference 150, 250, or 400 discussed herein above. In some embodiments, the operations are specifically performed by the information content based active learning module 117.

[00102] At reference 502, data points are obtained from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system. At reference 504, each of the data points is assigned, using a machine learning model, to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state. At reference 506, the data points are labeled to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

[00103] The array of state classes can be expressed as {US, NS, F F₂, ... , F_M} and the corresponding probabilities of the data points being at each state can be expressed as the label indicating if x belongs to class i, p(yt) is the predicted probability of x belonging to class i, and i E {US, NS, F_lt F₂, ... , F_M}. The one or more criteria are discussed herein above relating to the Section about the active learning measures of informativeness (e.g., formula (4) to (7)) and Figure 4. In some embodiments, the probability of a data point belongs to a state may be inferred from the probabilities of the data point belongs to other states without explicitly calculating the probability of the data point at the state - for example, when a data point belongs to any known state is below a threshold (e.g., 0.01), the probability of the data point belongs to the unknown state may be set to a high value (e.g., 0.9) without explicitly calculating the probability.

[00104] In some embodiments, the data points are received upon a detection of a concept drift within the cloud system.

[00105] In some embodiments, the machine learning model is retrained based on a probability of the data point being at a state class. For example, the machine learning model is retrained based on the probability of the data point being at a state class across a threshold for a corresponding binary entropy. The threshold may be set to thresholds c as discussed relating to Formula (4). Alternatively, the threshold may be set based on average binary entropy for a class over a number of metric data points as discussed relating to Formula (5).

[00106] Alternatively or additionally, the machine learning model is retrained based on the probability of the data point being at the unknown state class crossing a threshold in some embodiments. That threshold may be set based on Formula (6). The application of these thresholds is also shown in Formula (7) and Figure 4.

[00107] In some embodiments, retraining the machine learning model comprises running one or more test suites on the cloud system and mapping the one or more state classes to measurements of the one or more metrics are generated by running the one or more test suites. [00108] In some embodiments, the mapping of the one or more state classes to measurements of the one or more metrics are used to adjust parameters of the machine learning model. In some embodiments, the adjusted parameters of the machine learning model are used to assign data points that were assigned to the unknown state to either the normal state or one or more fault states.

[00109] In some embodiments, the machine learning model deploys a moving time window over a time series. In some embodiments, the machine learning model comprises a neural network (e.g., a long short-term memory network) that generates compressed feature representation of the moving time window over the time series as in the discussion about E(x) above.

[00110] In some embodiments, the machine learning model is trained using a set of pseudo negative data points generated from a uniform distribution, as shown in Figure 3.

[00111] In some embodiments, the labeled data points are transmitted to the cloud system to enhance performance of the cloud system.

[00112] Embodiments of the invention enable automated labeling of cloud system metrics, which is a challenging task to do manually, even for domain experts. These embodiments allow discovery of new patterns in the cloud system metrics that manifest into faults and obtain the corresponding labeled cloud system metrics, the discovery through informativeness measures is not feasible with prior fault injection mechanisms. Furthermore, the labeling may respond to concept drift and obtain corresponding labels through test suite running test cases and feed the labeling results to retrain the classifiers. Through running test cases to detect fault states of a cloud system and feeding the resulting labels to a classifier, the classifier may be trained to label all the detectable fault states. Such automated data labeling greatly reduces cloud resource consumption (since test suites are invoked only when needed) and human efforts required to label data points.

Devices Implementing Embodiments of the Invention

[00113] Figure 6 illustrates an electronic device to label data points of a cloud system automatically per some embodiments. The electronic device 602 may be a host in a cloud system, or a network node/UE in a wireless/wireline network, and the operating environment and further embodiments the host, the network node, the UE are discussed in more details herein below relating to Figures 7 to 12. The electronic device 602 may be implemented using custom application-specific integrated-circuits (ASICs) as processors and a special-purpose operating system (OS), or common off-the-shelf (COTS) processors and a standard OS. In some embodiments, the electronic device 602 implements a data point labeling system 655, which may operate all or subset of operations of the data point labeling system such as 150, 250, or 400.

[00114] The electronic device 602 includes hardware 640 comprising a set of one or more processors 642 (which are typically COTS processors or processor cores or ASICs) and physical network interfaces (NIs) 646, as well as non-transitory machine-readable storage media 649 having stored therein software 650. During operation, the one or more processors 642 may execute the software 650 to instantiate one or more sets of one or more applications 664A-R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment, the virtualization layer 654 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 662A-R called software containers that may each be used to execute one (or more) of the sets of applications 664A-R. The multiple software containers (also called virtualization engines, virtual private servers, or jails) are user spaces (typically a virtual memory space) that are separate from each other and separate from the kernel space in which the operating system is run. The set of applications running in a given user space, unless explicitly allowed, cannot access the memory of the other processes. In another such alternative embodiment, the virtualization layer 654 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the sets of applications 664A-R run on top of a guest operating system within an instance 662A-R called a virtual machine (which may in some cases be considered a tightly isolated form of software container) that run on top of the hypervisor - the guest operating system and application may not know that they are running on a virtual machine as opposed to running on a “bare metal” host electronic device, or through paravirtualization the operating system and/or application may be aware of the presence of virtualization for optimization purposes. In yet other alternative embodiments, one, some, or all of the applications are implemented as unikemel(s), which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library' operating system (LibOS) including drivers/libraries of OS services) that provide the particular OS services needed by the application. As a unikernel can be implemented to run directly on hardware 640, directly on a hypervisor (in which case the unikemel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented fully with unikemels running directly on a hypervisor represented by virtualization layer 654, unikemels running within software containers represented by instances 662A-R, or as a combination of unikemels and the above-described techniques (e.g., unikemels and virtual machines both run directly on a hypervisor, unikemels, and sets of applications that are run in different software containers).

[00115] The software 650 contains the data point labeling system 655 that performs operations described with reference to operations as discussed relating to Figures 1 to 5. The data point labeling system 655 may be instantiated within the applications 664 A-R. The instantiation of the one or more sets of one or more applications 664A-R, as well as virtualization if implemented, are collectively referred to as software instance(s) 652. Each set of applications 664A-R, corresponding virtualization construct (e.g., instance 662A-R) if implemented, and that part of the hardware 640 that executes them (be it hardware dedicated to that execution and/or time slices of hardware temporally shared), forms a separate virtual electronic device 660A-R. [00116] A network interface (NI) may be physical or virtual. In the context of Internet Protocol (IP), an interface address is an IP address assigned to an NI, be it a physical NI or virtual NI. A virtual NI may be associated with a physical NI, with another virtual interface, or stand on its own (e.g., a loopback interface, a point-to-point protocol interface). A NI (physical or virtual) may be numbered (a NI with an IP address) or unnumbered (a NI without an IP address). The NI is shown as network interface card (NIC) 644. The physical network interface 646 may include one or more antenna of the electronic device 602. An antenna port may or may not correspond to a physical antenna. The antenna comprises one or more radio interfaces.

A Wireless Network per Some Embodiments

[00117] Figure 7 illustrates an example of a communication system 700 per some embodiments.

[00118] In the example, the communication system 700 includes a telecommunication network 702 that includes an access network 704, such as a radio access network (RAN), and a core network 706, which includes one or more core network nodes 708. The access network 704 includes one or more access network nodes, such as network nodes 710A and 710B (one or more of which may be generally referred to as network nodes 710), or any other similar 3^rd Generation Partnership Project (3 GPP) access node or non-3GPP access point. The network nodes 710 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 712A, 712B, 712C, and 712D (one or more of which may be generally referred to as UEs 712) to the core network 706 over one or more wireless connections.

[00119] Example wireless communications over a wireless connection include transmitting and/or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and/or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 700 may include any number of wired or wireless networks, network nodes, UEs, and/or any other components or systems that may facilitate or participate in the communication of data and/or signals whether via wired or wireless connections. The communication system 700 may include and/or interface with any type of communication, telecommunication, data, cellular, radio network, and/or other similar type of system.

[00120] The UEs 712 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and/or operable to communicate wirelessly with the network nodes 710 and other communication devices. Similarly, the network nodes 710 are arranged, capable, configured, and/or operable to communicate directly or indirectly with the UEs 712 and/or with other network nodes or equipment in the telecommunication network 702 to enable and/or provide network access, such as wireless network access, and/or to perform other functions, such as administration in the telecommunication network 702.

[00121] In the depicted example, the core network 706 connects the network nodes 710 to one or more hosts, such as host 716. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 706 includes one more core network nodes (e.g., core network node 708) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and/or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 708. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and/or a User Plane Function (UPF).

[00122] The host 716 may be under the ownership or control of a service provider other than an operator or provider of the access network 704 and/or the telecommunication network 702, and may be operated by the service provider or on behalf of the service provider. The host 716 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio/video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[00123] As a whole, the communication system 700 of Figure 7 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and/or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and/or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and/or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[00124] In some examples, the telecommunication network 702 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunication network 702 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 702. For example, the telecommunication network 702 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and/or Massive Machine Type Communication (mMTC)/Massive loT services to yet further UEs.

[00125] In some examples, the UEs 712 are configured to transmit and/or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 704 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 704. Additionally, a UE may be configured for operating in single- or multiple radio access technology (multi-RAT) or multistandard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e., being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[00126] In the example, the hub 714 communicates with the access network 704 to facilitate indirect communication between one or more UEs (e.g., UE 712C and/or 712D) and network nodes (e.g., network node 710B). In some examples, the hub 714 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 714 may be a broadband router enabling access to the core network 706 for the UEs. As another example, the hub 714 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 710, or by executable code, script, process, or other instructions in the hub 714. As another example, the hub 714 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 714 may be a content source. For example, for a UE that is a virtual reality (VR) headset, display, loudspeaker or other media delivery device, the hub 714 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 714 then provides to the UE either directly, after performing local processing, and/or after adding additional local content. In still another example, the hub 714 acts as a proxy server or orchestrator for the UEs, in particular in if one or more of the UEs are low energy loT devices. [00127] The hub 714 may have a constant/persistent or intermittent connection to the network node 710B. The hub 714 may also allow for a different communication scheme and/or schedule between the hub 714 and UEs (e.g., UE 712C and/or 712D), and between the hub 714 and the core network 706. In other examples, the hub 714 is connected to the core network 706 and/or one or more UEs via a wired connection. Moreover, the hub 714 may be configured to connect to a machine-to-machine (M2M) service provider over the access network 704 and/or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 710 while still connected via the hub 714 via a wired or wireless connection. In some embodiments, the hub 714 may be a dedicated hub - that is, a hub whose primary function is to route communications to/from the UEs from/to the network node 710B. In other embodiments, the hub 714 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 710B, but which is additionally capable of operating as a communication start and/or end point for certain data channels.

UE per Some Embodiments

[00128] Figure 8 illustrates a UE 800 per some embodiments. As used herein, a UE refers to a device capable, configured, arranged and/or operable to communicate wirelessly with network nodes and/or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded/integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3 GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and/or an enhanced MTC (eMTC) UE.

[00129] A UE may support device-to-device (D2D) communication, for example by implementing a 3 GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehi cl e-to- vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle- to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and/or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[00130] The UE 800 includes processing circuitry 802 that is operatively coupled via a bus 804 to an input/output interface 806, a power source 808, a memory 810, a communication interface 812, and/or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 8. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[00131] The processing circuitry 802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 810. The processing circuitry 802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general -purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 802 may include multiple central processing units (CPUs).

[00132] In the example, the input/output interface 806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and/or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 800. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[00133] In some embodiments, the power source 808 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 808 may further include power circuitry for delivering power from the power source 808 itself, and/or an external power source, to the various parts of the UE 800 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 808. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 808 to make the power suitable for the respective components of the UE 800 to which power is supplied.

[00134] The memory 810 may be or be configured to include memory such as random-access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable readonly memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 810 includes one or more application programs 814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 816. The memory 810 may store, for use by the UE 800, any of a variety of various operating systems or combinations of operating systems.

[00135] The memory 810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a Universal Subscriber Identity Module (USIM) and/or IP Multimedia Services Identity Module (ISIM), other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 810 may allow the UE 800 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 810, which may be or comprise a device-readable storage medium.

[00136] The processing circuitry 802 may be configured to communicate with an access network or other network using the communication interface 812. The communication interface 812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 822. The communication interface 812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 818 and/or a receiver 820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 818 and receiver 820 may be coupled to one or more antennas (e.g., antenna 822) and may share circuit components, software or firmware, or alternatively be implemented separately.

[00137] In the illustrated embodiment, communication functions of the communication interface 812 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short- range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and/or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol/internet protocol (TCP/IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), Quick UDP Internet Connections (QUIC), Hypertext Transfer Protocol (HTTP), and so forth.

[00138] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 812, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[00139] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[00140] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door/window sensor, a flood/moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and/or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 800 shown in Figure 8. [00141] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and/or measurements and transmits the results of such monitoring and/or measurements to another UE and/or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and/or reporting on its operational status or other functions associated with its operation.

[00142] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g., by controlling an actuator) to increase or decrease the drone’s speed. The first and/or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

Network Node per Some Embodiments

[00143] Figure 9 illustrates a network node 900 per some embodiments. As used herein, network node refers to equipment capable, configured, arranged and/or operable to communicate directly or indirectly with a UE and/or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)). [00144] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units and/or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[00145] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi -standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell/multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and/or Minimization of Drive Tests (MDTs).

[00146] The network node 900 includes a processing circuitry 902, a memory 904, a communication interface 906, and a power source 908. The network node 900 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 900 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 900 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 904 for different RATs) and some components may be reused (e.g., a same antenna 910 may be shared by different RATs). The network node 900 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 900, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 900.

[00147] The processing circuitry 902 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and/or encoded logic operable to provide, either alone or in conjunction with other network node 900 components, such as the memory 904, to provide network node 900 functionality.

[00148] In some embodiments, the processing circuitry 902 includes a system on a chip (SOC). In some embodiments, the processing circuitry 902 includes one or more of radio frequency (RF) transceiver circuitry 912 and baseband processing circuitry 914. In some embodiments, the radio frequency (RF) transceiver circuitry 912 and the baseband processing circuitry 914 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 912 and baseband processing circuitry 914 may be on the same chip or set of chips, boards, or units. [00149] The memory 904 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and/or any other volatile or non-volatile, non-transitory device-readable and/or computer-executable memory devices that store information, data, and/or instructions that may be used by the processing circuitry 902. The memory 904 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and/or other instructions capable of being executed by the processing circuitry 902 and utilized by the network node 900. The memory 904 may be used to store any calculations made by the processing circuitry 902 and/or any data received via the communication interface 906. In some embodiments, the processing circuitry 902 and memory 904 is integrated.

[00150] The communication interface 906 is used in wired or wireless communication of signaling and/or data between a network node, access network, and/or UE. As illustrated, the communication interface 906 comprises port(s)/terminal(s) 916 to send and receive data, for example to and from a network over a wired connection. The communication interface 906 also includes radio front-end circuitry 918 that may be coupled to, or in certain embodiments a part of, the antenna 910. Radio front-end circuitry 918 comprises filters 920 and amplifiers 922. The radio front-end circuitry 918 may be connected to an antenna 910 and processing circuitry 902. The radio front-end circuitry may be configured to condition signals communicated between antenna 910 and processing circuitry 902. The radio front-end circuitry 918 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 918 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 920 and/or amplifiers 922. The radio signal may then be transmitted via the antenna 910. Similarly, when receiving data, the antenna 910 may collect radio signals which are then converted into digital data by the radio front-end circuitry 918. The digital data may be passed to the processing circuitry 902. In other embodiments, the communication interface may comprise different components and/or different combinations of components.

[00151] In certain alternative embodiments, the network node 900 does not include separate radio front-end circuitry 918, instead, the processing circuitry 902 includes radio front-end circuitry and is connected to the antenna 910. Similarly, in some embodiments, all or some of the RF transceiver circuitry 912 is part of the communication interface 906. In still other embodiments, the communication interface 906 includes one or more ports or terminals 916, the radio front-end circuitry 918, and the RF transceiver circuitry 912, as part of a radio unit (not shown), and the communication interface 906 communicates with the baseband processing circuitry 914, which is part of a digital unit (not shown).

[00152] The antenna 910 may include one or more antennas, or antenna arrays, configured to send and/or receive wireless signals. The antenna 910 may be coupled to the radio front-end circuitry 918 and may be any type of antenna capable of transmitting and receiving data and/or signals wirelessly. In certain embodiments, the antenna 910 is separate from the network node 900 and connectable to the network node 900 through an interface or port.

[00153] The antenna 910, communication interface 906, and/or the processing circuitry 902 may be configured to perform any receiving operations and/or certain obtaining operations described herein as being performed by the network node. Any information, data and/or signals may be received from a UE, another network node and/or any other network equipment. Similarly, the antenna 910, the communication interface 906, and/or the processing circuitry 902 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and/or signals may be transmitted to a UE, another network node and/or any other network equipment.

[00154] The power source 908 provides power to the various components of network node 900 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 908 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 900 with power for performing the functionality described herein. For example, the network node 900 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 908. As a further example, the power source 908 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[00155] Embodiments of the network node 900 may include additional components beyond those shown in Figure 9 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and/or any functionality necessary to support the subject matter described herein. For example, the network node 900 may include user interface equipment to allow input of information into the network node 900 and to allow output of information from the network node 900. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 900.

Host per Some Embodiments

[00156] Figure 10 is a block diagram of a host 1000, which may be an embodiment of the host 716 of Figure 7, per various aspects described herein. As used herein, the host 1000 may be or comprise various combinations hardware and/or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 1000 may provide one or more services to one or more UEs.

[00157] The host 1000 includes processing circuitry 1002 that is operatively coupled via a bus 1004 to an input/output interface 1006, a network interface 1008, a power source 1010, and a memory 1012. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 8 and 9, such that the descriptions thereof are generally applicable to the corresponding components of host 1000.

[00158] The memory 1012 may include one or more computer programs including one or more host application programs 1014 and data 1016, which may include user data, e.g., data generated by a UE for the host 1000 or data generated by the host 1000 for a UE. Embodiments of the host 1000 may utilize only a subset or all of the components shown. The host application programs 1014 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., Free Lossless Audio Codec (FLAC), Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1014 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1000 may select and/or indicate a different host for over-the-top services for a UE. The host application programs 1014 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.

Virtualization Environment per Some Embodiments

[00159] Figure 11 is a block diagram illustrating a virtualization environment 1100 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1100 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized.

[00160] Applications 1102 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 1100 to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein.

[00161] Hardware 1104 includes processing circuitry, memory that stores software and/or instructions executable by hardware processing circuitry, and/or other hardware devices as described herein, such as a network interface, input/output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1106 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1108a and 1108b (one or more of which may be generally referred to as VMs 1108), and/or perform any of the functions, features and/or benefits described in relation with some embodiments described herein. The virtualization layer 1106 may present a virtual operating platform that appears like networking hardware to the VMs 1108.

[00162] The VMs 1108 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1106. Different embodiments of the instance of a virtual appliance 1102 may be implemented on one or more of VMs 1108, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[00163] In the context of NFV, a VM 1108 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1108, and that part of hardware 1104 that executes that VM, be it hardware dedicated to that VM and/or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1108 on top of the hardware 1104 and corresponds to the application 1102.

[00164] Hardware 1104 may be implemented in a standalone network node with generic or specific components. Hardware 1104 may implement some functions via virtualization.

Alternatively, hardware 1104 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1110, which, among others, oversees lifecycle management of applications 1102. In some embodiments, hardware 1104 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1112 which may alternatively be used for communication between hardware nodes and radio units.

Communication among host, network node, and UE per Some Embodiments [00165] Figure 12 illustrates a communication diagram of a host 1202 communicating via a network node 1204 with a UE 1206 over a partially wireless connection per some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 712a of Figure 7 and/or UE 800 of Figure 8), network node (such as network node 710a of Figure 7 and/or network node 900 of Figure 9), and host (such as host 716 of Figure 7 and/or host 1000 of Figure 10) discussed in the preceding paragraphs will now be described with reference to Figure 12.

[00166] Like host 1000, embodiments of host 1202 include hardware, such as a communication interface, processing circuitry, and memory. The host 1202 also includes software, which is stored in or accessible by the host 1202 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1206 connecting via an over-the-top (OTT) connection 1250 extending between the UE 1206 and host 1202. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1250. [00167] The network node 1204 includes hardware enabling it to communicate with the host 1202 and UE 1206. The connection 1260 may be direct or pass through a core network (like core network 706 of Figure 7) and/or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.

[00168] The UE 1206 includes hardware and software, which is stored in or accessible by UE 1206 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1206 with the support of the host 1202. In the host 1202, an executing host application may communicate with the executing client application via the OTT connection 1250 terminating at the UE 1206 and host 1202. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1250 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1250. [00169] The OTT connection 1250 may extend via a connection 1260 between the host 1202 and the network node 1204 and via a wireless connection 1270 between the network node 1204 and the UE 1206 to provide the connection between the host 1202 and the UE 1206. The connection 1260 and wireless connection 1270, over which the OTT connection 1250 may be provided, have been drawn abstractly to illustrate the communication between the host 1202 and the UE 1206 via the network node 1204, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

[00170] As an example of transmitting data via the OTT connection 1250, in step 1208, the host 1202 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1206. In other embodiments, the user data is associated with a UE 1206 that shares data with the host 1202 without explicit human interaction. In step 1210, the host 1202 initiates a transmission carrying the user data towards the UE 1206. The host 1202 may initiate the transmission responsive to a request transmitted by the UE 1206. The request may be caused by human interaction with the UE 1206 or by operation of the client application executing on the UE 1206. The transmission may pass via the network node 1204, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1212, the network node 1204 transmits to the UE 1206 the user data that was carried in the transmission that the host 1202 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1214, the UE 1206 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1206 associated with the host application executed by the host 1202.

[00171] In some examples, the UE 1206 executes a client application which provides user data to the host 1202. The user data may be provided in reaction or response to the data received from the host 1202. Accordingly, in step 1216, the UE 1206 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input/output interface of the UE 1206. Regardless of the specific manner in which the user data was provided, the UE 1206 initiates, in step 1218, transmission of the user data towards the host 1202 via the network node 1204. In step 1220, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1204 receives user data from the UE 1206 and initiates transmission of the received user data towards the host 1202. In step 1222, the host 1202 receives the user data carried in the transmission initiated by the UE 1206.

Terms

[00172] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” and so forth, indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[00173] The description and claims may use the terms “coupled” and “connected,” along with their derivatives. These terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of wireless or wireline communication between two or more elements that are coupled with each other. A “set,” as used herein can refer to any whole number of items including one item.

[00174] An electronic device (such as the electronic device 602) stores and transmits (internally and/or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as a computer program code or a computer program) and/or data using machine-readable media (also called computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine- readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical, or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., of which a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), other electronic circuitry, or a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and/or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code/data even when the electronic device is turned off (when power is removed). When the electronic device is turned on, that part of the code that is to be executed by the processor(s) of the electronic device is typically copied from the slower non-volatile memory into volatile memory (e.g., dynamic random-access memory (DRAM), static random-access memory (SRAM)) of the electronic device. Typical electronic devices also include a set of one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and/or receive code and/or data using propagating signals) with other electronic devices. For example, the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, or translating to allow the electronic device to send and receive data whether over a wired and/or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of (1) receiving data from other electronic devices over a wireless connection and/or (2) sending data out to other devices through a wireless connection. This radio circuitry may include transmitter(s), receiver(s), and/or transceiver(s) suitable for radio frequency communication. The radio circuitry may convert digital data into a radio signal having the proper parameters (e.g., frequency, timing, channel, bandwidth, and so forth). The radio signal may then be transmitted through antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate with wire through plugging in a cable to a physical port connected to an NIC. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and/or hardware. [00175] The terms “module,” “logic,” and “unit” used in the present application, may refer to a circuit for performing the function specified. In some embodiments, the function specified may be performed by a circuit in combination with software such as by software executed by a general -purpose processor.

[00176] Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and/or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional unit to perform corresponding functions according one or more embodiments of the present disclosure.

[00177] The term unit may have conventional meaning in the field of electronics, electrical devices, and/or electronic devices and may include, for example, electrical and/or electronic circuitry, devices, modules, processors, memories, logic solid state and/or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and/or displaying functions, and so on, as such as those that are described herein.

EMBODIMENTS

1. A method to be implemented in an electronic device, the method comprising: obtaining (502) data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning (504), using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling (506) the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria. 2. The method of embodiment 1, wherein the data points are received upon a detection of a concept drift within the cloud system.

3. The method of embodiment 1 or 2, wherein the machine learning model is retrained based on a probability of the data point being at a state class.

4. The method of any of embodiments 1 to 3, wherein the machine learning model is retrained based on the probability of the data point being at a state class crosses a threshold for a corresponding binary entropy.

5. The method of any of embodiments 1 to 4, wherein the machine learning model is retrained based on the probability of the data point being at the unknown state across a threshold.

6. The method of any of embodiments 1 to 5, wherein retraining the machine learning model comprises running one or more test suites on the cloud system and mapping the one or more state classes to measurements of the one or more metrics are generated by running the one or more test suites.

7. The method of any of embodiments 1 to 6, wherein the mapping of the one or more state classes to measurements of the one or more metrics are used to adjust parameters of the machine learning model.

8. The method of any of embodiments 1 to 7, wherein the adjusted parameters of the machine learning model are used to assign data points that were assigned to the unknown state to either the normal state or one of the one or more fault states.

9. The method of any of embodiments 1 to 8, wherein the machine learning model deploys a moving time window over a time series.

10. The method of any of embodiments 1 to 9, wherein the machine learning model comprises a neural network that generates compressed feature representation of the moving time window over the time series.

11. The method of any of embodiments 1 to 10, wherein the machine learning model is trained using a set of pseudo negative data points generated from a uniform distribution.

12. The method of any of embodiments 1 to 11, wherein the labeled data points are transmitted to the cloud system to enhance performance of the cloud system. 13. An electronic device (602), comprising: a processor (642) and non-transitory machine-readable storage medium (649) that provides instructions that, when executed by the processor (642), are capable of causing the processor (642) to perform: obtaining (502) data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning (504), using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling (506) the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

14. The electronic device of embodiment 13, wherein the data points are received upon a detection of a concept drift within the cloud system.

15. The electronic device of embodiment 13 or 14, wherein the machine learning model is retrained based on a probability of the data point being at a state class.

16. The electronic device of any of embodiments 13 to 15, wherein the machine learning model is retrained based on the probability of the data point being at a state class crosses a threshold for a corresponding binary entropy.

17. The electronic device of any of embodiments 13 to 16, wherein the machine learning model is retrained based on the probability of the data point being at the unknown state across a threshold.

18. The electronic device of any of embodiments 13 to 17, wherein retraining the machine learning model comprises running one or more test suites on the cloud system and mapping the one or more state classes to measurements of the one or more metrics are generated by running the one or more test suites.

19. The electronic device of any of embodiments 13 to 18, wherein the mapping of the one or more state classes to measurements of the one or more metrics are used to adjust parameters of the machine learning model. 20. The electronic device of any of embodiments 13 to 19, wherein the adjusted parameters of the machine learning model are used to assign data points that were assigned to the unknown state to either the normal state or one of the one or more fault states.

21. The electronic device of any of embodiments 13 to 20, wherein the machine learning model deploys a moving time window over a time series.

22. The electronic device of any of embodiments 13 to 21, wherein the machine learning model comprises a neural network that generates compressed feature representation of the moving time window over the time series.

23. The electronic device of any of embodiments 13 to 22, wherein the machine learning model is trained using a set of pseudo negative data points generated from a uniform distribution.

24. The electronic device of any of embodiments 13 to 23, wherein the labeled data points are transmitted to the cloud system to enhance performance of the cloud system.

25. A machine-readable storage medium (649) that provides instructions that, when executed by a processor, are capable of causing the processor to perform any of methods 1 to 12.

26. A computer program that provides instructions that, when executed by a processor, are capable of causing the processor to perform any of methods 1 to 12.

Claims

CLAIMS What is claimed is:

1. A method to be implemented in an electronic device, the method comprising: obtaining (502) data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning (504), using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling (506) the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

2. The method of claim 1, wherein the data points are received upon a detection of a concept drift within the cloud system.

3. The method of claim 1 or 2, wherein the machine learning model is retrained based on a probability of the data point being at a state class.

4. The method of any of claims 1 to 3, wherein the machine learning model is retrained based on the probability of the data point being at a state class crosses a threshold for a corresponding binary entropy.

5. The method of any of claims 1 to 4, wherein the machine learning model is retrained based on the probability of the data point being at the unknown state across a threshold.

6. The method of any of claims 1 to 5, wherein retraining the machine learning model comprises running one or more test suites on the cloud system and mapping the one or more state classes to measurements of the one or more metrics are generated by running the one or more test suites.

7. The method of any of claims 1 to 6, wherein the mapping of the one or more state classes to measurements of the one or more metrics are used to adjust parameters of the machine learning model.

8. The method of any of claims 1 to 7, wherein the adjusted parameters of the machine learning model are used to assign data points that were assigned to the unknown state to either the normal state or one or more fault states.

9. The method of any of claims 1 to 8, wherein a moving time window is deployed over a time series to obtain the data points.

10. The method of any of claims 1 to 9, wherein the machine learning model comprises a neural network that generates compressed feature representation of the moving time window over the time series.

11. The method of any of claims 1 to 10, wherein the machine learning model is trained using a set of pseudo negative data points generated from a uniform distribution.

12. The method of any of claims 1 to 11, wherein the labeled data points are transmitted to the cloud system to enhance performance of the cloud system.

13. An electronic device (602), comprising: a processor (642) and non-transitory machine-readable storage medium (649) that provides instructions that, when executed by the processor (642), are capable of causing the processor (642) to perform: obtaining (502) data points from a cloud system, each data point including measurements of one or more metrics to track operational status of the cloud system; assigning (504), using a machine learning model, each of the data points to an array of state classes including a normal state, one or more fault states, and an unknown state, the assignment of a data point being mapped to corresponding probabilities of the data points being at each state; and labeling (506) the data points to one or more states within the array of state classes when probabilities of the data points meet one or more criteria.

14. The electronic device of claim 13, wherein the data points are received upon a detection of a concept drift within the cloud system.

15. The electronic device of claim 13 or 14, wherein the machine learning model is retrained based on a probability of the data point being at a state class.

16. The electronic device of any of claims 13 to 15, wherein the machine learning model is retrained based on the probability of the data point being at a state class crosses a threshold for a corresponding binary entropy.

17. The electronic device of any of claims 13 to 16, wherein the machine learning model is retrained based on the probability of the data point being at the unknown state across a threshold.

18. The electronic device of any of claims 13 to 17, wherein retraining the machine learning model comprises running one or more test suites on the cloud system and mapping the one or more state classes to measurements of the one or more metrics are generated by running the one or more test suites.

19. The electronic device of any of claims 13 to 18, wherein the mapping of the one or more state classes to measurements of the one or more metrics are used to adjust parameters of the machine learning model.

20. The electronic device of any of claims 13 to 19, wherein the adjusted parameters of the machine learning model are used to assign data points that were assigned to the unknown state to either the normal state or one of the one or more fault states.

21. The electronic device of any of claims 13 to 20, wherein a moving time window is deployed over a time series to obtain the data points.

22. The electronic device of any of claims 13 to 21, wherein the machine learning model comprises a neural network that generates compressed feature representation of the moving time window over the time series.

23. The electronic device of any of claims 13 to 22, wherein the machine learning model is trained using a set of pseudo negative data points generated from a uniform distribution.

24. The electronic device of any of claims 13 to 23, wherein the labeled data points are transmitted to the cloud system to enhance performance of the cloud system.

25. A machine-readable storage medium (649) that provides instructions that, when executed by a processor, are capable of causing the processor to perform any of methods 1 to 12.

26. A computer program that provides instructions that, when executed by a processor, are capable of causing the processor to perform any of methods 1 to 12.