EP4732196A1

EP4732196A1 - Federated learning hyperparameter optimization

Info

Publication number: EP4732196A1
Application number: EP24743192.7A
Authority: EP
Inventors: Quang Ly; Charles AYDIN; Dale Seed; Lu Liu
Original assignee: InterDigital Patent Holdings Inc
Current assignee: InterDigital Patent Holdings Inc
Priority date: 2023-06-26
Filing date: 2024-06-26
Publication date: 2026-04-29
Also published as: WO2025006581A1; CN121399623A

Abstract

Described herein are methods for advanced federated learning hyperparameter optimization. Federated learning hyperparameter optimization presents challenges to machine-learning training that do not exist in centralize hyperparameter optimization. Issues such as lack of data visibility, model parameter aggregation strategies, communications overhead between federated learning servers and federated learning clients, federated learning training round schedules, and availability of federated learning clients need to be addressed. Certain enhancements may be made to federated learning training to improve the federated learning hyperparameter optimization process. This disclosure proposes the following enhancements: a method for configuring hyperparameters for federated learning training; a method for requesting federated learning hyperparameter optimization search space reduction; and a method for configuring federated learning hyperparameter optimization training rounds.

Description

FEDERATED LEARNING HYPERPARAMETER OPTIMIZATION

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to, and the benefit of the filing date of, U.S. provisional patent application no. 63/510,298, filed June 26, 2023, the contents of which are incorporated by reference herein in their entirety.

BACKGROUND

[0002] Although hyperparameter optimization is well-explored in the centralized machine-learning context, hyperparameter optimization in federated learning is relatively new and introduces unique challenges and opportunities compared to the centralized training scenario. The distributed nature of federated learning necessitates communication-efficient optimization methods, as updating hyperparameters across all devices can be resource and communication intensive. Furthermore, heterogeneity in local data and device capabilities can result in varying optimal hyperparameters, which is referred to as the personalization aspect of federated learning. In addition, privacy plays a crucial role in federated learning due to which a centrally managed hyperparameter optimization approach may not be feasible and/or efficient. Last but not the least, varying degrees of limited computational resources (computing power, storage, or network bandwidth) of the federated learning clients may affect their performance and scalability, thus limiting their ability to perform hyperparameter optimization operations efficiently.

[0003] Federated learning hyperparameter optimization presents challenges to both computing costs as well as communication costs that differs from centralize hyperparameter optimization. In centralized hyperparameter optimization, it is typical to orchestrate virtual machines in the cloud or data centers without worrying about the schedule or availability of the clients performing the training. In addition, since a central dataset is shared among the virtual machines for training, the data scientist has a general idea about the data distribution and quality of the data used for training. In federated learning hyperparameter optimization, this is not the case, and the federated learning user is not exposed to the data quality and/or data distribution provided by each of the federated learning clients. The availability of federated learning clients is also out of the control of the federated learning server/user and hence, communications between the federated learning server and federated learning clients becomes a factor in federated learning training. The combination of federated learning client availability and the unknown data quality and distribution of the datasets from the perspective of the federated learning user make federated learning hyperparameter optimization inherently more difficult than centralize hyperparameter optimization.

[0004] Another aspect of federated learning hyperparameter optimization that makes it more difficult than centralize hyperparameter optimization is the need to perform aggregation of model parameters from different federated learning clients during each training round. The choice of aggregation strategy of the model parameters may impact the federated learning hyperparameter optimization results and thus must be accounted for whereas this is not a concern with centralize hyperparameter optimization.

SUMMARY

[0005] Described herein are methods for advanced federated learning hyperparameter optimization. Federated learning hyperparameter optimization presents challenges to machinelearning training that do not exist in centralized hyperparameter optimization. Issues such as lack of data visibility, model parameter aggregation strategies, communications overhead between federated learning servers and federated learning clients, federated learning training round schedules, and availability of federated learning clients need to be addressed. Certain enhancements may be made to federated learning training to improve the federated learning hyperparameter optimization process. This disclosure proposes the following enhancements: a method for configuring hyperparameters for federated learning training; a method for requesting federated learning hyperparameter optimization search space reduction; and a method for configuring federated learning hyperparameter optimization training rounds.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to features that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The following detailed description is better understood when read in conjunction with the appended drawings. For the purposes of illustration, examples are shown in the drawings; however, the subject matter is not limited to specific elements and instrumentalities disclosed. In the drawings:

[0008] FIG. 1 shows an example of a machine-learning lifecycle;

[0009] FIG. 2 shows an example of hyperparameter optimization;

[0010] FIG. 3 shows an example of exploratory data analysis output examples for numerical and categorical data;

[0011] FIG. 4 shows an example of exploratory data analysis output histogram examples;

[0012] FIG. 5 shows an example of a scatter plot of exploratory data analysis output and a correlation matrix related to the features of the scatter plot;

[0013] FIG. 6A shows an example environment;

[0014] FIG. 6B shows an example method of federated learning training and hyperparameter configuration requests;

[0015] FIG. 7 shows an example method of federated learning hyperparameter search space reduction;

[0016] FIGs. 8A, 8B, and 8C together show an example of a subsampling dataset comparison to whole machine-learning datasets;

[0017] FIG. 9 shows an example method of federated learning scheduled hyperparameter optimization;

[0018] FIG. 10 shows an example of a federated learning training schedule;

[0019] FIG. 11 shows an example of a graphical user interface for a federated learning hyperparameter optimization dynamic configuration status;

[0020] FIG. 12 shows an example of a RESTful resource embodiment;

[0021] FIG. 13 shows an example of a graphical user interface for a federated learning training request;

[0022] FIG. 14 shows an example of a graphical user interface for a federated learning configuration request;

[0023] FIG. 15A shows an example communications system; [0024] FIG. 15B shows another example communication system;

[0025] FIG. 15C shows another example communication system;

[0026] FIG. 15D shows another example communication system;

[0027] FIG. 15E shows another example communications system;

[0028] FIG. 15F shows an example apparatus or device; and

[0029] FIG. 15G shows an example computing system.

DETAILED DESCIPTION

[0030] Described herein are methods for advanced federated learning hyperparameter optimization. This disclosure proposes the following enhancements: a method for configuring hyperparameters for federated learning training; a method for requesting federated learning hyperparameter optimization search space reduction; and a method for configuring federated learning hyperparameter optimization training rounds.

[0031] A method for a federated learning service to receive a first request to configure federated learning training may offer a first enhancement. The request may comprise one or more of: a federated learning user identifier, machine-learning application, machine-learning algorithms, machine-learning function types, target feature, minimum number of features, feature list, minimum number of federated learning clients, dataset size requirements, data coverage and distribution requirements, target completion time, and federated learning client capabilities. The federated learning service may send one or more exploratory data analysis requests to candidate federated learning clients. The federated learning service may receive responses to the exploratory data analysis requests. The responses may comprise one or more of: dataset identifiers, general summary of the features for a dataset, tabular information such as unique value and missing value counts, numerical data statistics, histogram and scatter plots of features, correlation matrix and plots between features, and federated learning client capabilities. The federated learning service may create a client pool for federated learning training. The client pool may be comprised of federated learning clients that can perform federated learning training and have available datasets that may be used in the federated learning training. The federated learning service may send a response to the first request. The response may comprise one or more of: a federated learning training identifier, number of federated learning clients, anonymized list of federated learning client identifiers, dataset identifiers and associate feature list, dataset size, data coverage percentage and distribution, machine-learning algorithm identifiers, hyperparameter optimization algorithms, hyperparameter configurations, hyperparameters, range and increment size of numerical hyperparameters, and a list of values for categorical hyperparameters.

[0032] A method for a federated learning user to send a first request to configure federated learning training may offer another enhancement. The first request may comprise one or more of a federated learning user identifier, machine-learning application, machine-learning algorithms, machine-learning function types, target feature, minimum number of features, feature list, minimum number of federated learning clients, dataset size requirements, data coverage and distribution requirements, target completion time, and federated learning client capabilities. The federated learning user may receive a notification of exploratory data analysis query results, the exploratory data analysis results comprise one or more of dataset identifiers, general summary of the features for a dataset, tabular information such as unique value and missing value counts, numerical data statistics, histogram and scatter plots of features, correlation matrix and plots between features, and federated learning client capabilities. The federated learning user may send a response to the notification. The response may indicate whether a dataset from a federated learning client can be used in federated learning training. The federated learning user may receive a response to the first request, the response may comprise one or more of: a federated learning training identifier, number of federated learning clients, anonymized list of federated learning client identifiers, dataset identifiers and associate feature list, dataset size, data coverage percentage and distribution, machine-learning algorithm identifiers, hyperparameter optimization algorithms, hyperparameter configuration, hyperparameters, range and increment size of numerical hyperparameters, and a list of values for categorical hyperparameters.

[0033] A method for a federated learning service to receive a first request to configure hyperparameters for federated learning training may offer yet another enhancement. The request may comprise one or more of: a federated learning training identifier, machine-learning algorithm and associated hyperparameters, hyperparameter ranges, an allowed tuning indicator, minimum number of federated learning clients, dataset size requirements, data coverage and distribution requirements, target completion time, and a selection of federated learning client identifiers and datasets for the federated learning training. The federated learning service may derive hyperparameter configurations based on the request parameters in the first request, including new hyperparameter sets not specified by the request parameters if the allowed tuning indicator is provided to the federated learning service. The federated learning service may send a response to the first request. The response may comprise one or more of federated learning training identifier, the derived hyperparameter configurations, the total number of federated learning clients, a list of anonymized federated learning clients and associate dataset, federated learning client dataset size and data coverage/distribution percentages, the total dataset size, the expected number of training rounds, and the estimated completion time.

[0034] Another enhancement may be achieved in a method for a federated learning user to send a first request to configure hyperparameters for federated learning training. The request may comprise one or more of: federated learning training identifier, machine-learning algorithm and associated hyperparameters, hyperparameter ranges, an allowed tuning indicator, minimum number of federated learning clients, dataset size requirements, data coverage and distribution requirements, target completion time, and a selection of federated learning client identifiers and datasets for the federated learning training. The federated learning user may receive a response to the first request. The response may comprise one or more of: a federated learning training identifier, the derived hyperparameter configurations, the total number of federated learning clients, a list of anonymized federated learning clients and associate dataset, federated learning client dataset size and data coverage/distribution percentages, the total dataset size, the expected number of training rounds, and the estimated completion time.

[0035] Yet another enhancement may be found in a method for a federated learning service to receive a first request to perform federated learning hyperparameter search space reduction. The request may comprise one or more of: federated learning training identifier, federated learning user identifier, hyperparameter search space, target search space reduction percentage, target performance percentage, aggregation strategies, minimum number of federated learning clients, minimum number of training rounds, minimum data partition size, and minimum data coverage and distribution. The federated learning service may send one or more dataset sub-sampling requests to candidate federated learning clients, the requests comprise one or more of: a dataset identifier, minimum data partition size, and a minimum data coverage and distribution for the partitioned dataset. The federated learning service may receive responses to the subsampling requests, the response comprise one or more of: a partitioned dataset identifier, the partitioned dataset size, the data coverage and distribution of the partitioned dataset, and an expiration for the availability of the partitioned dataset. The federated learning service may send one or more exploratory data analysis requests to the federated learning clients to perform on the partitioned dataset. The federated learning service may receive exploratory data analysis results from the one or more federated learning clients, the exploratory data analysis results are obtained from performing exploratory data analysis on the partitioned dataset. The federated learning service may send a response to the first request. The response may comprise one or more of federated learning training identifier; federated learning hyperparameter optimization identifier; statistics associated with each hyperparameter configuration such as performance metric and reason for early stop; recommendations of hyperparameter configurations to pursue for the actual federated learning hyperparameter optimization; hyperparameter search space reduction; partitioned dataset identifier, size, coverage, and distribution; a list of the participating federated learning clients; the total number of training rounds; and the total elapse time of the training.

[0036] A method for a federated learning user to send a first request to perform federated learning hyperparameter search space reduction may offer yet another enhancement. The request may comprise one or more of a federated learning training identifier, federated learning user identifier, hyperparameter search space, target search space reduction percentage, target performance percentage, aggregation strategies, minimum number of federated learning clients, minimum number of training rounds, minimum data partition size, and minimum data coverage and distribution. A federated learning user may receive one or more notifications of exploratory data analysis results obtained from partitioned datasets of federated learning clients. A federated learning user may send acknowledgements to the federated learning service with indications of suitable partitioned datasets for use in FL hyperparameter search space reduction training. A federated learning user may receive a notification from the federated learning service. The notification may comprise one or more of: a federated learning training identifier; federated learning hyperparameter optimization identifier; statistics associated with each hyperparameter configuration such as performance metric and reason for early stop; recommendations of hyperparameter configurations to pursue for the actual federated learning hyperparameter optimization; hyperparameter search space reduction; partitioned dataset identifier, size, coverage, and distribution; a list of the participating federated learning clients; the total number of training rounds; and the total elapse time of the training. [0037] Another enhancement may be achieved in a method for a federated learning service to receive a first request to configure a federated learning hyperparameter optimization training round. The request may comprise one or more of: a federated learning training identifier, federated learning HPO identifier, federated learning user identifier, an automated training indicator, machine-learning algorithm/model to train, hyperparameter optimization algorithms, hyperparameter configurations, aggregation strategies, minimum number of federated learning clients, a list of federated learning clients and associated dataset identifier, minimum number of training rounds, minimum dataset size per training round, model parameter update intervals, and target completion time. A federated learning service may send a response to the first request. The response may comprise one or more of: a federated learning training identifier, federated learning HPO identifier, machine-learning algorithm/model to train, hyperparameter configurations, aggregation strategies, total number of federated learning clients, a list of anonymized federated learning clients and associate dataset, federated learning client dataset size and data coverage/distribution percentages, the aggregate dataset size, the expected number of training rounds, model parameter update intervals, and an estimated completion time. A federated learning service may schedule federated learning hyperparameter optimization training on the one or more federated learning clients. The schedule may comprise one or more of: training and training round identifiers, training round details, model information and parameters, list of HP configurations and the associated dataset identifiers, data size for the training round, training time requirements, and model parameter update intervals. A federated learning service may send notifications to update the federated learning user about the progress of the federated learning hyperparameter optimization training. The notification may comprise one or more of: federated learning training identifier, federated learning hyperparameter optimization identifier, list of the best performing machine-learning models, statistics associated with each hyperparameter configuration, a list of the participating federated learning clients; dataset identifiers with corresponding metrics for size, coverage and distribution percentages, aggregate dataset size, the total number of training rounds, the total elapse time of the training, and recommendations of additional hyperparameter configurations for future federated learning hyperparameter optimization runs.

[0038] A method for a federated learning user to send a first request to configure a federated learning hyperparameter optimization training round may offer yet another enhancement. The request may comprise one or more of: a federated learning training identifier, federated learning hyperparameter optimization identifier, federated learning user identifier, an automated training indicator, machine-learning algorithm/model to train, hyperparameter optimization algorithms, hyperparameter configurations, aggregation strategies, minimum number of federated learning clients, a list of federated learning clients and associated dataset identifier, minimum number of training rounds, minimum dataset size per training round, model parameter update intervals, and target completion time. A federated learning user may receive a response to the first request. The response may comprise one or more of: a federated learning training identifier, federated learning hyperparameter optimization identifier, machine-learning algorithm/model to train, hyperparameter configurations, aggregation strategies, total number of federated learning clients, a list of anonymized federated learning clients and associate dataset, federated learning client dataset size and data coverage/distribution percentages, the aggregate dataset size, the expected number of training rounds, model parameter update intervals, and an estimated completion time. A federated learning user may receive notifications to update the federated learning user about the progress of the federated learning hyperparameter optimization training. The notification may comprise one or more of: federated learning training identifier, federated learning hyperparameter optimization identifier, list of the best performing ML models, statistics associated with each hyperparameter configuration, a list of the participating federated learning clients; dataset identifiers with corresponding metrics for size, coverage and distribution percentages, aggregate dataset size, the total number of training rounds, the total elapse time of the training, and recommendations of additional hyperparameter configurations for future federated learning hyperparameter optimization runs.

[0039] The following abbreviations, terms, and definitions may be used herein:

[0040] Machine Learning (ML) is a branch of computer science that empowers computer systems to "learn" from data (i.e., continually enhance performance on predicting relevant metrics) without explicit programming. The training of an ML model or system involves optimizing the parameters of an ML algorithm or a collection of algorithms, comparing the prediction outcomes with the actual historical data of the target metric(s), based on the corresponding historical data of their drivers (features). The following is a general overview of the complete ML lifecycle, as depicted in FIG. 1.

[0041] The ML lifecycle is a cyclical process that commences with grasping the business requirements and dynamics, which subsequently informs the understanding of the data. Following this, the data is prepared for the training and validation of an ML model, which is then deployed and assessed against the objectives set by the business requirements. Based on these requirements, an ML application is established, and its data needs are determined. An ML pipeline is subsequently constructed, where data collection and preparation take place for the ML application. The processed data is then fed into an ML model for training, validation, and verification, ensuring that the model satisfies the business requirements. Once the model is trained, it is deployed for inference, and its performance is monitored to guarantee the inference results align with the ML application's goals. If necessary, new data features may be collected, further data preparation may be employed and/or new algorithms may be explored to update the ML model, should it produce poor inferencing results, which may indicate model/data drift.

[0042] An integral part of the training process is the hyperparameter optimization (HPO) process, as shown in FIG. 2. In ML, both model parameters and hyperparameters (HP) play critical roles in achieving effective and efficient models. Parameters are the model's internal values that are learned and updated during training to minimize a loss function. Examples of model parameters comprise model weights and biases in a neural network. On the other hand, hyperparameters are external configurations that are set prior to training and remain constant throughout the training process. Examples of hyperparameters comprise learning rate, batch size, and number of layers in a neural network. While model parameters directly influence the model's performance on training data, hyperparameters help determine the overall model architecture and the training process, ultimately improving the model's ability to generalize well to unseen data.

[0043] Similar to regular model parameter optimization obtained from training a machine learning algorithm, HPO aims to find the optimal set of hyperparameters that yield the best model performance on validation data. Various optimization techniques have been proposed, including grid search, random search, Bayesian optimization, etc. Grid search involves an exhaustive search over a predefined hyperparameter search space, while random search samples hyperparameters from a defined distribution. Bayesian optimization models the objective function as a Gaussian process, iteratively refining the search based on previous evaluations. More advanced techniques like genetic algorithms and particle swarm optimization have also been applied.

[0044] FL is an approach to ML that enables the creation of models on data that is distributed across multiple clients or locations, without the need to aggregate and store data in one central location. An FL server manages multiple FL clients during training and/or inferencing and data is kept private and stored locally at the individual FL clients. ML model(s) and the associated parameters are exchanged between the FL server and the FL clients during training. The FL server aggregates the model parameters (e.g., weights) at each round of federated training and the FL training iterates for many rounds. This method has become increasingly popular in recent years as more and more organizations seek to harness the power of large datasets without compromising the privacy of individual users and/or data sources, with the added benefits of reduced data movement, and improved scalability. However, FL also introduces several challenges, one of which is hyperparameter optimization.

[0045] EDA is a fundamental step in the ML process, providing a crucial foundation for any data-driven solution. It is the process of summarizing, visualizing, and interpreting the information that is hidden in the data. The primary purpose of EDA is to extract significant insights from data, understand the underlying structure, extract important variables, detect outliers and anomalies, complete data cleaning steps, test underlying assumptions, and develop parsimonious models. EDA employs a variety of statistical and graphical techniques to maximize the data scientist’s insight into a dataset’s structure, anomalies, patterns, and relationships. It is a preparatory step that aids in the ML model selection and helps data scientists anticipate the challenges of data preprocessing, such as handling missing or outlier values. By doing so, EDA not only strengthens the accuracy of ML models but also provides a direction for the general ML pipeline.

[0046] EDA may comprise steps to examine the correlations, distribution/variability of features, numerical data statistics, among other aspects of a dataset to provide insights into the quality of the dataset. In the below examples, various EDA outputs are presented for a particular dataset showing both numerical and categorical data and the corresponding EDA outputs. FIG. 3 shows examples of EDA outputs such as the total number of data instances, the number of features in the dataset, the number of unique data values for each feature of the dataset, the number of missing values for each feature of the dataset, and statistical measurements (e g. mean, standard deviation, etc.) of numerical data. Note that numerical data are represented with “nvar” prefix while categorical data are represented with “cvar” prefix.

[0047] The unique value counts output may be more useful for categorical features and may be used by the FL user to determine if all categories for a particular feature are covered by the dataset - it is assumed in this scenario that the FL user has general knowledge of the feature and knows of the possible outcomes. For numerical features, the unique value counts output may provide information about the coverage of the feature if the range of values are known by the FL user.

[0048] The missing value counts is another EDA output that applies to both numerical and categorical features. The output may identify the completeness of the values for a particular feature by comparing the number of missing values with the total number of data instances in the dataset. A feature with a high percentage of missing values may not contribute much to training the ML model and therefore, may be removed from the dataset especially if the missing values are consistently found across many datasets from different FL clients.

[0049] For numerical features, statistic outputs may provide additional information on the spread of values within the range of the particular feature. Statistics such as mean and standard deviation may show the data distribution of values while minimum and maximum values may show the range in which the values fall within. There may also be quartile percentages (e.g. 25%, 50%, and 75%) that further shows the data distribution of the values that the FL user and/or service may use to determine the coverage and/or quality of the dataset.

[0050] Another useful EDA output may be the histogram of a feature, namely a graph that shows how a feature’s values are distributed in a dataset. FIG. 4 shows an example of a histogram for a target variable (which may be a discrete numerical variable such an integer) of a particular dataset. The target variable may represent the labeled data of the dataset and the histogram may show the concentration of labels for the given dataset. The histogram output may also be applied to categorical data in which case the counts may represent the number of times a certain categorical value appears in the dataset. Note that the same information appearing in a histogram may also be obtained by querying the value counts of a variable, which may output a numerical table similar to the ones shown in FIG. 3. The histogram output may provide a visual representation of the distribution of the values of a feature to allow a FL user the ability to quickly examine the coverage and frequency of the value.

[0051] The distribution shown in FIG. 4 is centered around the mode (the most frequently occurring value) of the target variable, which is 14. The median, which is the boundary of the two halves of the data is 24, and the mean is roughly 37. This is a gamma like distribution where the distribution is skewed from the mean. The FL user may examine histogram outputs of various datasets to determine if the desired distribution of labels is satisfactory for training the ML model.

[0052] Correlation refers to the statistical relationship or association between two numerical variables. It measures the strength and direction of the linear relationship between the variables. Correlation is commonly used to determine how changes in one variable are related to changes in another variable. A high correlation, represented by a numerical value close to 1, shows the two variables has a strong relationship with each other while a low correlation value shows little relationship between the two variables.

[0053] Similarly, scatter plots are also widely used in data science as a visual tool to understand the relationship between two numerical variables. They provide a graphical representation of the data points, where each point corresponds to a specific combination of values from the two variables being plotted. For example, the correlation coefficient value of 0.8768 between the variables nvar_2 and nvar_3 also depicts itself in the scatter plot between these two variables (e.g., at the intersection of nvar_2 on the x-axis with nvar_3 on the y-axis). This signals the collinearity between the two variables, which means they have an inherent dependence on each other. In this case, it may be best to eliminate one of the two variables from the dataset as a feature, and only use one. Finally, the diagonal figures in the scatter plots are simply histograms of the corresponding variables, as the real scatter plot of a variable against itself would be a 45-degree line. There are other aspects of EDA that may help the FL user make decisions for data preparation and feature engineering to produce accurate ML models such as addressing data imbalances and missing data.

[0054] Correlation among features may be another important aspect to consider which is a measure for the inter-relationship between the features. This can be designated by a correlation matrix and/or scatter plots as shown in FIG. 5. [0055] A hyperparameter configuration is a unique combination of HP values. To illustrate; if learning rate Ir, number of epochs E and fraction of clients participating in FL fr are all hyperparameters to tune a given ML model training, the following would be one such HP configuration: Ir = 0.001, E = \Q,fr = 0.2. Such configurations may be specified by the FL user or they may be derived by an FL service. Other HP configurations may also be specified for FL HPO, such as Ir = 0.0015, E = 10, = 0.2, Ir = 0.001, E = 10, = 0.25, Ir = 0.001, E = 8,# = 0.2.

[0056] The set of HP configurations may be considered as the HP search space. Each HP configuration may require the training of an ML model to completion using the associated hyperparameters in the HP configuration. For example, the HP configuration [Ir = 0.001, E = 10, fr = 0.2] may require the training of an ML model where the learning rate is set at 0.001 and 20% of available FL clients are used for the training, which spans 10 epochs. Similar training may be performed for the other HP configurations listed above.

[0057] With federated learning, the training may further involve numerous communication exchanges between the FL service and FL clients where model parameters are exchanged for both the global model parameter download at the beginning of an FL training round and the local model parameter upload at the completion of an FL training round.

[0058] FL HPO necessitates the coordination and exchange of hyperparameter configurations and HPO processing steps between FL users and FL clients. An FL service can facilitate this process while also carrying out other FL-related tasks and operations. As a preliminary step, FL clients may have already set up communication channels with the FL service and may have shared essential client capabilities and accessible datasets for FL training.

[0059] In subsequent subsections, FL specific HPO elements and scenarios will be considered. Please note that the first pre-requisite step before any ML training is data preparation. Since HPO is logistically part of the ML training process, data preparation steps may need to be performed prior to the start of FL training. In this disclosure, it is assumed that the data preparation process has been completed or that the FL clients participating in FL training have the capability to perform the data preparation step prior to using the data for training.

[0060] In federated learning, there may be a large number of clients with available datasets in the FL network, in which case it may be more practical to select a subset of FL clients as participants. Therefore, the fraction of clients participating in the FL HPO process may be considered a hyperparameter. A related hyperparameter is the sampling algorithm for determining the sub-selection of available clients. The selection of a subsampling of clients can be done in different ways. A completely random selection is the easiest method. However, EDA based approaches may result in more representative samples. The following table depicts a more extensive list of plausible FL specific HPs and criteria for FL HPO.

[0061] A main part of federated learning is the aggregation strategy of model parameters at the end of each FL training round. Various aggregation strategies may be specified, such as pure averaging, weighted averaging, Bayesian, quantization, momentum, etc. Due to the differences in aggregation strategy, it may be considered as a hyperparameter for FL HPO as different strategies may produce different training results.

[0062] Other FL specific considerations may be considered hyperparameters as well since they may also impact training. These may comprise the number of training rounds, the number of local training iterations, the number of FL clients per training round, and/or the data coverage and distribution of the datasets available on FL clients.

[0063] The data distribution, which may also comprise data variability and/or data coverage, of the datasets available on FL clients may be an important consideration in FL HPO as it may indicate the quality of the data used in FL training. This information may be especially important as the FL user does not have visibility into the data and thus is not able to determine the quality or suitability of the data for FL training. Providing an FL user the ability to specify a requirement on the minimum percentage of data coverage and distribution for a dataset used in FL training may provide the FL user control over the selection of the datasets (and hence the FL clients) for the FL HPO process. Therefore, the data distribution may be considered as a FL hyperparameter.

[0064] The variability and/or coverage of a data may refer to whether all known values for a variable are found in the dataset for the feature, while the data distribution may refer to how many times each value appears in the dataset. For categorical variables, this may be designated by the number of unique values per variable and their frequencies. For numerical features, it may be the distribution of the values (for instance, the histogram) within a range between the minimum and maximum values for the feature. As a result, an FL user may use the data coverage and distribution of a target feature to determine if the necessary labels are present to train the ML model. Therefore, the variability and/or coverage of data may be considered as a FL hyperparameter.

[0065] The performance metric and convergence threshold information elements may also be considered as FL hyperparameters due to their impacts in the overall hyperparameter optimization process. The performance metric may inform the FL service what the minimum performance requirements are in order to be ranked as best performing hyperparameter configurations and/or trained models. Similarly, the convergence threshold also impacts the hyperparameter optimization process by providing the FL service another criterion for determining best performing hyperparameter configurations and models. As federated learning is divided into many training rounds, the convergence threshold may be utilized to track the convergence of the performance for each hyperparameter configurations. Thus, the convergence threshold may provide an early-termination criteria to stop training of HP configurations that falls under the convergence threshold between training rounds.

[0066] FIG. 6A shows an example environment in which the methods and apparatus described and claimed herein may be implemented. As shown in FIG. 6A, the environment may comprise a network 600, an FL service, one or more FL user devices (e.g., FL User 1, FL User 2, ... FL UserN), and one or more FL clients (e.g., FL Client 1, FL Client 2, ... FL Client N). The FL service may comprise a service implemented on a device, gateway, or server configured to communicate over the network 600. The FL service may be implemented as part of a cloudbased environment, as part of an edge network, or as part of application servers connected to the network 600. The network 600 may comprise any wired or wireless network, including for example, the internet and a mobile operator network. FL user devices may comprise client applications that send requests for FL HPO to the FL service over the network 600. These client applications may be used to provision and configure the FL service to perform FL HPO operations. The clients applications of the FL user devices may operated by data scientists who wish to find the best hyperparameters to train one or more machine learning models. The FL user devices on which these client applications reside may be WRTU or UE devices, application functions with access to the FL service, or devices that present graphical user interfaces for interacting with the FL service, or any other suitable computing device that can be used to communicate with the FL service. FL clients may comprise client applications that have registration links to the FL service and perform operations for FL HPO. The client applications may reside on mobile and non-mobile devices that provide data used as part of FL HPO and FL training, or any other suitable compute device capable of performing the functions of the FL client. The FL clients may collect data and participate in FL operations.

[0067] Prior to the start of FL HPO, an FL user may need to configure an FL service with FL training information. During this process, the FL user may also specify HP configurations for the FL HPO training.

[0068] FIG. 6B shows an example method where an FL user may first configure an FL service with FL training requirements and then specify HP configurations for the FL HPO. The FL user may be a novice user who may require guidance from an FL service on recommended hyperparameters for the FL training. The FL user may have a ML application in mind but does not have data to train a ML model. As a result, the FL user may make a request to configure FL training in which requirements are provided for the ML application. In response, the FL service may create a pool of FL clients that may have available data for the ML application and return a response specifying the list of anonymized FL clients and other details for the FL training. The FL service may include in the response, recommendations of hyperparameters and the associated range of values that may be tuned for the FL training.

[0069] If the FL user is more knowledgeable about the FL HPO training process, then steps 2 - 5 of FIG. 6B may be skipped and the FL user may only need to perform steps 6 - 8 of FIG. 6B to configure the hyperparameters for the FL HPO training. In this scenario, the FL user may also need to provide the requirements for the ML application (e.g. the request parameters specified in step 2) and the FL service may create the FL client pool as part of the FL HP configuration request.

[0070] In FIG. 6B, at step 1, an FL service may be made aware of FL clients and the available datasets (and associated information about the datasets) on the FL clients. The FL service may have been pre-provisioned with the FL client information, or the FL service may have received the information via FL client registration with the FL service. Other information about the FL clients may also be made available to the FL service, such as the FL client’s capabilities, operating environments, schedules, available datasets, etc. It is worth noting that the FL service may be pre-configured with information required to initiate communications with the FL clients or vice versa.

[0071] At step 2, an FL user may send a request to the FL service to indicate a need for federated training for a particular ML application. As shown in the table below, the request may comprise a FL training identifier, the ML application or application type, one or more candidate ML algorithms for training, a target feature, a minimum number of features, a list of feature names, minimum number of FL clients, size of the required dataset, data coverage and distribution requirements, target training completion time, FL client capabilities, or other parameters. Note the terms “features” and “variables” are used interchangeably in the following table and elsewhere in the disclosure to represent input data that may impact the ML application. In tabular form, the features or variables are the data found in the columns of the dataset. They represent the independent variables used in training the ML model.

[0072] At step 3, the FL service may create a RESTful resource for the request and store the request parameters provided by the FL user locally within the RESTful resource. Alternatively and/or additionally, the information may be stored in a database or some other storage mechanism. A FL training identifier may be assigned to the resource and returned to the FL user in a response message, e.g. in step 5. The FL user may use the FL training identifier in future requests to refer to the configuration information stored in the resource.

[0073] The FL service may then perform EDA requests to gather information from the FL clients about data coverage and distributions of features and target variables for each of the clients’ datasets. At the same time, the FL service may also request updates on the FL client capabilities such as client availability and schedule, new or updated datasets, and new or updated functional capabilities (e.g. hardware upgrades, operating system and software upgrades such as libraries and packages). This information may be used for determining the FL training configuration as well as configuring the hyperparameters. FL clients, even those with similar data (EDA) characteristics, may have different capabilities (compute, storage, network etc.), which may limit their participation in the FL training. The FL service may share the EDA results with the FL user to receive input from the FL user on dataset quality and/or FL clients the FL user may be interested in for FL training. Each available dataset in a FL client may be assigned an identifier to associate with the EDA results. The input from the FL user may assist the FL service in creating a pool of clients that may contribute to the FL training.

[0074] The EDA results from each of the FL clients may also be saved within the resource created for the FL training associated with the FL user. These EDA results may serve as the baseline EDA results for a dataset/FL client as it captures the data characteristics of the original dataset.

[0075] At step 4, the FL service, using information the FL service has obtained about the FL clients and their datasets, may determine whether FL clients can fulfill the FL requirements provided by the FL user. The FL service may evaluate the user requirements against the FL client information and group the FL clients that can participate in the FL training into a FL pool to train the one or more candidate ML algorithms. At the same time, the FL service may determine certain hyperparameters that may be applicable to the one or more ML algorithms and the range of values to associate with each of the hyperparameters.

[0076] At step 5, the FL service may provide a response to the FL training request with the results of its evaluation. The response may comprise a status to the training request, a FL training identifier, and other information listed in the following table. The FL service may assign a unique FL training identifier the FL user can reference for future requests associated with the FL training. For FL client related information, the FL service may provide the number of FL clients having the capability and datasets to participate in the FL training and an anonymized list of FL client identifiers with the corresponding information about the characteristics of the datasets to provide an indication of the quality of the datasets to the FL user. In addition to the FL client information, the FL service may also comprise suggestions on hyperparameters and the range of values for the hyperparameters that may be used in the FL training. The hyperparameters and the associated ranges may be organized according to the ML algorithm the FL user had specified. Additionally, the FL service may also recommend other ML algorithms that may also fulfill the goals of the ML application and the associated hyperparameters. The following table provides an example of the various information elements that may be included in the response to the FL training request.

[0077] At step 6, the FL user may evaluate the information provided by the FL service and determine the FL clients (and corresponding datasets) and/or hyperparameters to use for the FL training. The FL user may select to use the specified (by the FL service) range of hyperparameter values or limit the range further to reduce the HP search space used during FL training. The FL user may send the information in a request to the FL service to indicate the selection of FL clients, the FL training identifier, the ML algorithm and hyperparameters, the associated hyperparameter ranges for the FL training, and other information as specified in the tables above such as the aggregation strategy and performance metrics. The FL user may also provide an allowed tuning indication that allows the FL service to derive additional hyperparameter configurations from the ones that the FL user has provided, should a set of HP configurations not converge to a desired model accuracy sufficiently quickly/closely. For example, if the FL user had limited the range of some hyperparameter values, the indication may allow the FL service to select a hyperparameter value that is outside the range specified by the FL user but within the range that the FL service had suggested in step 5. Similarly, if the FL user had removed one of the hyperparameters from the suggested list, the indicator may allow the FL service to incorporate the hyperparameter into the HPO search space, albeit with one or two values for the hyperparameter.

[0078] Alternatively, or in addition, the functionality of requests 2 and 6 may be combined into a single request. In such a scenario, the FL user may have prior knowledge of and/or familiarity with the ML use case and the clients’ datasets. Note that the request parameters from step 2 (e.g. found in the tables above) and step 6 may be combined into the single request. One such example may be hospitals running a ML use case for disease diagnosis and each hospital wanting to keep their data private. The administrators of the hospital, who may serve the role of a FL user, may be aware of the ML use case and the fact that federated learning is enabled for the hospital to train a ML model with their data. The administrator may be aware of or have knowledge of the FL training and make one request to configure the FL training and the associated FL hyperparameter search space.

[0079] At step 7, the FL service may derive the hyperparameter configurations based on the FL user selection of hyperparameters and hyperparameter range values. If the allowed tuning indicator was provided by the FL user in step 6, the FL service may derive new HP configurations beyond what the FL user had specified. The FL service may also check the combinations of hyperparameters and hyperparameter values to ensure a suitable hyperparameter search space is defined. The FL service may associate the HP configurations to the FL training identifier.

[0080] At step 8, the FL service may return the set of HP configurations to the FL user, including the FL training identifier and the FL service derived HP configurations beyond what the FL user had selected for hyperparameters and hyperparameter values. Other information may al so be comprised, such as the total number of FL clients, a list of anonymized FL clients and associate dataset, FL client dataset size and data coverage/distribution percentages, the total dataset size, the expected number of training rounds, and the estimated completion time.

[0081] Although FIG. 6B shows an FL user making a FL HP configuration request after configuring an FL training request, other sequences of requests are also possible. In certain scenarios, a FL user may be knowledgeable about the FL HPO process such that the FL user may only need to perform the FL HP configuration request. In another scenario, a FL user may first configure a FL training request to obtain the available HP search space and then proceed to perform a FL HP search space reduction request to reduce the HP search space further. In another scenario, the user may send a FL HP search space reduction request after receiving the HP configuration response in step 8. The FL HP search space reduction method will be described in the disclosure hereinafter.

[0082] FL HPO has a further constraint compared to centralize ML HPO due to the federated nature of the training. FL client availability and computing capabilities may vary from client to client and an FL service must manage that aspect of FL HPO as well. As a result, it may be beneficial to reduce the HP search space as much as possible to minimize the time that it may take to complete the FL HPO. An approach to reducing the HP search space is to perform a “FL HP search space reduction” method prior to the actual FL HPO. In this method, the FL service may be able to perform an HPO on a reduced percentage of the dataset with preselected FL clients to determine a sub-set of the HP search space for use in the actual, “full” HPO training. The idea is that the HP search space reduction would be carried out on a sufficiently representative subset of FL client datasets in terms of data coverage and distributions to determine a reduced list of best HP configurations for use in the actual FL HPO.

[0083] FIG. 7 illustrates an example method for FL HP search space reduction. FIG. 7 presents an example scenario in which the FL service may be able to instruct FL clients to subsample their local data to form data partitions with sufficiently representative data distributions for use in the HP search space reduction method. This may help make the HP search space reduction method more efficient as less (but sufficiently representative) data would be used for the training. The resulting output of the FL HP search space reduction method may be used to reduce the HPO search space. [0084] As an example, an FL user may have already performed the FL hyperparameter configuration method as described in FIG. 6B to obtain the list of hyperparameter configurations that may be used for FL training. The FL user may determine that the hyperparameter search space is too large and it would take a long time to complete the FL training. Instead of using the entire hyperparameter search space, the FL user may request the FL service to perform a HP search space reduction method to reduce the hyperparameter search space. In another embodiment, the FL user may be familiar with FL HPO and have independently determined the hyperparameter search space but is interested in further reducing the number of hyperparameter configurations. The FL user in this case may also utilize the FL HP search space reduction method to try to further reduce the hyperparameter search space. Note also that the FL HP search space reduction method may be integrated with the FL HP configuration method described in FIG. 6B. For example, the FL service may provide a FL HP search space reduction indicator that an FL user may select in step 6 of FIG. 6B to inform the FL service to perform the method for HP search space reduction described hereinafter as part of FL HP configuration.

[0085] As one example, a method for federated learning (FL) hyperparameter (HP) search space reduction may comprise receiving, by a FL service configured to communicate via a network, from a FL user device that has requested HP optimization in connection with a FL training of a machine learning model, a request to reduce a HP search space. The request to reduce the HP search space may comprise an identifier associated with the FL training and one or more requirements associated with reduction of the HP search space. The FL service may then send, to each of a plurality of FL clients, a request to partition a dataset of the FL client to create a subsampling of the dataset of the FL client for use in HP search space reduction. The FL service may then create a configuration for HP search space reduction training by the plurality of FL clients based on the one or more requirements received in the request to reduce the HP search space. The FL service may then cause the HP search space reduction training to be performed by the plurality of FL clients based on the configuration. The FL service may then receive, from each of the plurality of FL clients, a result of the HP search space reduction training performed by the FL client. The result may indicate a performance associated with one or more HP configurations. The FL service may send, to the FL user device, information indicative of the results of the HP search space reduction training performed by the plurality of FL clients. The information may be sent in the form of a message and/or presented via a graphical user interface of the FL user device. The information may comprise one or more of: the identifier associated with the FL training of the machine learning model; a list of the best performing HP configurations resulting from the HP search space reduction training; statistics associated with each HP configuration; or one or more recommendations of HP configurations to pursue in connection with FL HP optimization.

[0086] The one or more requirements associated with reduction of the HP search space comprise one or more of: an indication of a target percentage reduction in the HP search space; an indication of a maximum number of HP configurations for the HP search space; an indication of a target percentage for minimizing a loss function or maximizing accuracy associated with the HP optimization; an indication of one or more aggregation strategies for use during the HP search space reduction; an indication of a minimum number of FL clients to be employed for HP search space reduction; an indication of a minimum number of training rounds required for HP search space reduction; an indication of a size of a data partition of a dataset to be used for HP search space reduction; or an indication of minimum data coverage or distribution percentage for a particular feature in a data partition of a dataset to be used for HP search space reduction.

[0087] Each request to partition a dataset of one of the plurality of FL clients may comprise a dataset identifier, an indication of a minimum size of a partition of the dataset of the FL client, and an indication of a minimum data coverage and distribution.

[0088] The FL service may further be configured to create a RESTful resource for storing at least (i) the one or more requirements received in the request from the FL user device for HP search space reduction, and (ii) information associated with the partitioned datasets of each of the plurality of FL clients. The FL service may assign an identifier to the RESTful resource, and may send, to the FL user device, the identifier of the RESTful resource.

[0089] The configuration for HP search space reduction training may comprise one or more of: a schedule for the HP search space reduction training by the plurality of FL clients; identifiers associated with one or more rounds of training; information associated with each round of training; identifiers of the partitioned datasets of each FL client; parameters associated with the machine learning model; information associated with one or more HP configurations; or an expiration time associated with the training.

[0090] The FL service may further be configured to determine, based upon the results of the HP search space reduction training performed by each of the FL clients, a ranking of the performance of each of the HP configurations. The FL service may reduce a number of the one or more HP configurations based on the one or more requirements of the request received from the FL user device.

[0091] FIG. 7 illustrates an example method for federated learning (FL) hyperparameter (HP) search space reduction in greater detail. As shown in FIG. 7, at step 1, FL users may request an FL service to assist with the reduction of the hyperparameter search space by sending a FL HP search space reduction request. The request may comprise the FL training identifier and other information listed in the table below. The FL user may provide requirements for the FL HP search space reduction method such as indications of the minimum number of FL clients, the aggregation strategies to use, the minimum number of data instances in each subsampled data partition, and the minimum data coverage/distribution for the datasets. In addition, the FL user may provide the HP search space and indications of the target reduction percentage and/or target performance percentage for the FL HP search space reduction. The target reduction percentage and/or target performance may additionally and/or alternatively be provided by local configuration and/or local policies of the FL service.

[0092] At step 3, the FL service may send requests to FL clients to partition their dataset for use in the method. The request may comprise a dataset identifier, minimum data partition size, and a minimum data coverage and distribution so that the dataset partition may represent a subsampling of the original dataset. The number or percentage of data instances in the partition may be specified by the FL user or determined by the FL service. Interactions between the FL service and the FL clients may be iterative until the FL clients have obtained a partitioned dataset that meets the requirements specified in the request. The FL clients may respond to the FL service with a partitioned dataset identifier, the partitioned dataset size, the data coverage and distribution of the partitioned dataset, and an expiration for the availability of the partitioned dataset. [0093] As part of this step, the FL service may perform EDA requests for the partitioned dataset to obtain the coverage and distribution of the data and the selection of FL clients for the HP search space reduction training (which may depend on the EDA outcomes). The FL service may also need to ensure other requirements as specified by the FL user in the table above are met. At the end of this step, partitioned dataset may be available for the HP search space reduction training. The partitioned dataset may represent a sub sampling (e.g. 10%) of the original dataset but with a similar data coverage and distribution as the original dataset, for which the EDA results may have been previously obtained (e.g. by step 3 of FIG. 6B). The FL service may perform iterative EDA requests with the FL clients to derive the partitioned dataset to meet the desired requirements. The FL service may expose the EDA results (e.g. via notifications) to the FL user to receive feedback on the EDA results of the subsampled dataset partitions. The FL user may indicate whether a partitioned dataset is suitable for training based on the EDA results when acknowledging the FL service. Similarly, the FL service may determine whether a partitioned dataset is suitable by comparing the EDA results of the partitioned dataset with the EDA results of the original dataset (e.g. as obtained by step 3 of FIG. 6B). This step may therefore be interactive with multiple iterations between the FL service, FL clients, and FL users.

[0094] The subsampled dataset partitions may be used in the HP search space reduction training and should have a similar data coverage and distribution as the original dataset. Correlation information and distribution information from the EDA results can be used as measurements of the similarity between the subsampled dataset partitions and the original dataset. Mathematical similarities of correlation matrices may be computed (e.g., Spearman’s or Kendall’s rank correlations) as a single numerical measure for this purpose. For comparing individual metrics’ distributions, correlation coefficients between histograms (i.e., vectors of bin counts) may be calculated as well. FIGs. 8 A, 8B, and 8C show examples of EDA results for two subsampled datasets compared to an original dataset - FIG. 8A represents the original or whole dataset and FIGs. 8B and 8C represents the good and bad, respectively, subsampled datasets. FIG. 8B is an example of a “good dataset” that was subsampled from the original dataset based on the distribution of the target variable with three other numerical variables. The correlation information is also very similar to the original dataset which is a good sign in terms of a distribution match. Effectively, the probability density function (PDF) of the values of the target variable was applied to the original dataset using a sampling percentage of 10%. When trained using the original dataset, the model showed an R-squared value of about 0.735 (a value of 1 would be an exact match of prediction values to real values). With the “good” subsampled dataset, the R-squared only drops to about 0.716. This means with a smaller, albeit “representative” subsample dataset, a ML model with similar accuracy (as trained with the original dataset) can be obtained by training the model with the subsampled dataset.

[0095] On the other hand, FIG. 8C represents a “bad” subsampled dataset that was obtained via quasi-random subsampling. The distribution of the target variable with the three numerical variables for the “bad” subsampled dataset does not closely match the distribution of the original dataset, especially for nvar_2 and nvar_4. Similarly, the correlation values of the “bad” subsampled dataset are quite different from that of the original dataset and the model accuracy drop is extensive - from 0.735 to 0.585.

[0096] At step 3 of FIG. 7, the FL service may create a RESTful resource and store the request parameters provided by the FL user in step 1 and the partitioned dataset information and EDA results from all the FL clients locally within the RESTful resource. Alternatively and/or additionally, the information may be stored in a database or some other storage mechanism. A FL HPO identifier may be assigned to the resource and returned to the FL user in a response message with the contents of the resource. The FL user may use the FL HPO identifier in future requests to refer to the configuration information stored in the resource.

[0097] At step 4, the FL user or FL service may configure HP search space reduction training based on the requirements provided by the FL user in step 1 and any new configurations the FL user may provide for this step. The FL service may need to schedule the HP search space reduction training of the selected FL clients in an efficient manner to minimize communication overhead as well as data distributions of the datasets, e.g. to schedule datasets with larger data distribution earlier in the training process. The configuration may comprise: training and training round identifiers, training round details, partitioned dataset identifier, model information and parameters, list of HP configurations, model parameter update intervals, and an expiration time for the training.

[0098] At step 5, the FL service may then schedule the training rounds with each of the FL clients for the FL HP search space reduction training based on the configurations determined in step 4. To minimize communication overhead, the FL service may schedule the training relative to training round data size in which FL clients may cycle through each of the HP configurations using the same data for a particular training round.

[0099] At step 6, each FL client may perform the scheduled training round configured by the FL service using the sub sampled dataset from step 2. As part of the HP search space reduction training, the FL clients may cycle the same subsampled data through each of the HP configurations to minimize communications overhead.

[00100] At step 7, the completion of the training round, the FL clients may update model parameters for each HP configuration. The training results may be communicated from the FL clients to the FL service at the completion of each training round, as indicated by the training schedule, or on demand by a FL service request. The FL service may perform the appropriate aggregation strategy with training results from all the FL clients associated with a particular HP configuration and may calculate the corresponding performance metric for the HP configuration. The FL service may also be able to determine the convergence rate of the performance metric to verify if the training is improving the model.

[0101] At step 8, the FL service may evaluate the training results at the end of each training round and may keep a list of best performing HP configurations. As the training rounds progress, the FL service may be able to determine certain HP configurations are not converging towards the target performance metric and therefore may be subject to early stop termination. The FL service may generate notes for the early stop of a HP configuration that may later be shared with the FL user when reporting the results. Steps 4 to 8 may be repeated for multiple training rounds until all the data of the subsampled dataset have been exhausted. The FL service may also expose the real time updates of the training results to the FL user and the FL user may be allowed to perform early stop termination of certain underperforming HP configurations.

[0102] At step 9, at the end of the HP search space reduction training, the FL service may evaluate the performance of all the HP configurations and determine a ranking of the best performing HP configurations. The FL service may perform a reduction of the HP configurations according to the criteria provided by the FL user or provided by local configuration and/or policy. For example, the FL service may select only those HP configurations that satisfy the criteria provided by the FL user.

[0103] At step 10, the FL service may share the results of the HP search space reduction training with the FL user and provide a list of the best performing HP configurations in a notification message sent to the FL user or via a GUT on a dashboard display. Included in the notification message may be the FL training identifier and the FL HPO identifier. Statistics associated with each HP configuration may also be provided in the notification message to provide more information of other metrics the FL service may have maintained during the training. For example, the FL service may provide performance metrics, trend lines of the accuracy function through the various training rounds, and the reason for the early stop of certain HP configurations. The FL service may provide recommendations of HP configurations to pursue for the actual FL HPO and other information that may also be comprised in the notification message are the HP search space reduction; partitioned dataset identifier, size, coverage, and distribution; a list of the participating FL clients; the total number of training rounds; and the total elapse time of the training.

[0104] The methods shown in FIG. 6B and FIG. 7 may offer FL users control over the FL HPO process in configuring FL training, configuring FL HP configurations, and obtaining FL HP search space reductions before performing the actual FL HPO training. Note that the three separate requests may be integrated into a single request with various options a FL user can select to enable certain part of the requests. For example, the FL training request may comprise options for enabling HP configuration as well as HP search space reduction methods. A FL user may select the appropriate options and let the FL service manage the FL HP search space and the FL HPO process. The options may also be presented to a FL user through a GUI display.

[0105] The FL HPO process can be made more efficient by introducing a dynamic monitoring and/or intervention capability. Initially, a FL user may have obtained HP configurations from the FL service (e.g. from the FL configuration requests described in FIG. 6B or through the FL HP search space reduction method described in FIG. 7) or the FL user may proactively determine the HP configurations for example based on prior knowledge and/or experience of the FL HPO process.

[0106] The user may then send a FL monitoring request to the FL service to receive notifications of updates from the FL service on the status of the HPO process such as training round completion status, number of training rounds completed, quantified model performance, auto-generation status of new HP configurations, etc. Subsequently, the FL user may configure the FL service to initiate the FL HPO training by sending a request to the FL service to configure the training round automatically, manually, and/or interactively. Training may be scheduled such that each client may perform training of multiple HP configurations for each round using the same data to minimize communication overhead between the FL clients and the FL service.

[0107] FIG. 9 shows a method for such dynamic monitoring and/or intervention capability. At step 1, it is assumed that the FL user has performed one of the previously described methods: FL training configuration, FL HP configuration, or FL HP search space reduction. Thus, the FL user may have configured the FL service with the FL HP configurations (e.g., the HP search space) as described in FIG. 6B or have obtained reduced HP configurations as described in FIG. 7 and is ready to proceed with the actual FL HPO training.

[0108] The FL user may perform each of the methods in whatever order the FL user may deem necessary. For example, a FL user may configure FL training and then request FL HP search space reduction prior to requesting FL HPO. In another example, the FL user may only want to request FL training and let the FL service manage the HP search space and the FL HPO training. In a third example, the FL user may request FL HP configuration after receiving unsatisfactory results from an initial FL HPO training.

[0109] At step 2, the FL user may make a FL monitoring request to be notified of certain events during the FL HPO process. For example, some events the FL user may want to be notified of are at the completion of each training round, after a number of training rounds, when the performance of a HP configuration has achieved a certain level, when a HP configuration has been removed from FL HPO training, if a new HP configuration has been derived and enabled for training, after some time duration, etc. The FL user may provide the FL training identifier and the FL HPO identifier in the request to associate the FL HPO training with the appropriate configurations.

[0110] At step 3, the FL service may respond with an acknowledgement to the monitoring request from the FL user, the response may comprise a status to the request and a monitoring ID the FL service may comprise in future notifications the FL service may send to the FL user.

[OHl] At step 4, the FL user may configure the FL service to enable the start of FL HPO training and provide the information presented in the table below in a request to the FL service to configure a FL HPO training round. The request may configure the FL service to automate FL HPO training by including an automate training indicator, which may specify that the FL service may manage the entire FL HPO training process without further intervention from the FL user. Alternatively, the FL user may want to control the FL HPO training process and exclude the automate training indicator. The FL user may then interactively manage the FL HPO training.

[0112] The request may comprise information that the FL service may need to manage the FL HPO training, such as the HP configurations, the aggregation strategy to use, the minimum number of FL clients, the minimum number of training rounds, and the total aggregate dataset size for the training. An FL user identifier may be comprised to associate the FL user with the FL HPO training. Contact information may be maintained by the FL service in case the FL service needs to inform the FL user of issues during the FL HPO training or to send notifications to the FL user.

[0113] Using the information provided by the FL user, the FL service may determine (or estimate) the total number of training rounds required to complete the FL training, the number of data instances required for each round of training, and the frequency of model parameter updates required from the FL clients for each combination of HP configuration and aggregation strategy. The FL service may store this information locally in a RESTful resource associated with the FL HPO and may return the information contained in the resource to the FL user. The resource may contain information such as the FL training identifier, FL HPO identifier, ML algorithm/model to train, HP configurations, aggregation strategies, total number of FL clients, a list of anonymized FL clients and associate dataset, FL client dataset size and data coverage/distribution percentages, the aggregate dataset size, the expected number of training rounds, model parameter update intervals, and an estimated completion time. Alternatively and/or additionally, the information may be stored in a database or some other storage mechanism.

[0114] At step 5, the FL service may schedule the FL HPO training based on the information provided by the FL user. To minimize communication overhead, the FL service may schedule FL HPO training in which FL clients cycle through each of the HP configurations while using the same training data to train the model for a training round. This predetermined schedule (either provided by the user or automatically by the service) may be sent to FL clients as part of a one-time configuration message or it may be performed in real time during the start of each FL training round.

[0115] FIG. 10 shows an example of scheduled FL HPO training. The figure shows that the FL HPO process may be performed with a combination of HP configurations and different aggregation strategies. The schedule may divide the FL HPO training into separate training rounds and may indicate when local model parameters may be uploaded to the FL service as represented by triangles in the figure. If FL HP search space reduction method was previously run, the FL service may schedule the FL HPO training rounds to continue from where it left off. This may reduce the time necessary to complete the FL HPO training as e.g. 10% of the dataset have already been trained. The FL service may maintain a real time computation of the performance metric for each combination of HP configuration and aggregation strategy.

[0116] For each training round, the same data may be used to train each of the HP configuration and aggregation strategy combination to minimize communication overhead between the FL clients and the FL service. At the indicated model parameter upload points, the FL clients may upload the model parameters for each of the HP configuration and the FL service may perform individual aggregation strategy of the model parameters from all FL clients for a particular HP configuration and aggregation strategy combination. Note the information in FIG. 10 represents the perspective of the FL service and what the FL service may expose to the FL user of the scheduled FL HPO training. The aggregation strategy may only be visible within the FL service and may be opaque to the FL clients. Note also that a schedule may exist for each FL client participating in the FL HPO training.

[0117] When configuring the FL clients, the FL service may provide training and training round identifiers, training round details, dataset identifier, model information and parameters, list of HP configurations, data size for the training round, processing time requirements, and when to report model updates (e.g. after completion of each HP configuration and aggregation strategy combination).

[0118] As shown in FIG. 9, at step 6, the FL clients may perform the scheduled FL HPO training provided by the FL service using the same data for the training round for each HP configuration. As a result, the FL clients may have model parameter updates for each of the HP configuration at the end of a training round or as indicated by the FL HPO training schedule. The FL clients may maintain model parameters for each HP configuration and aggregation strategy combination. Since the FL clients may not be aware of aggregation strategy, the FL service may configure the FL clients with a separate indicator. For example and using FIG. 10 as a basis, the FL service may configure HP configurations 2 and 3 as 2a, 2b, 3a, and 3b where a and b indicate the different aggregation strategy associated with the corresponding HP configurations.

[0119] At step 7, at the completion of each FL HPO training round, the FL clients may update model parameters for each HP configuration. Fhe training results may be communicated from the FL clients to the FL service at the completion of each training round, as indicated by the FL HPO training schedule, or on demand by a FL service request. The FL service may perform the appropriate aggregation strategy with training results from all the FL clients associated with a particular HP configuration and may calculate the corresponding performance metric for the HP configuration. The FL service may also be able to determine the convergence rate of the performance metric to verify if the training is improving the model.

[0120] At step 8, if the FL user had requested for a FL HPO monitoring of training round completion event, the FL service may send a notification to the FL user indicating the completion of a training round. The FL service may also provide real-time status for each HP configuration, the performance metric associated with each HP configuration, the percentage of FL HPO training completion, the remaining number of training rounds, the elapsed time of the FL HPO training, etc., in a GUI dashboard presented to the FL user or as notifications sent to the FL user.

[0121] The FL service may evaluate the training results at the end of each training round and may keep a list of best performing HP configurations. As the training rounds progress, the FL service may be able to determine certain HP configurations are not converging towards the target performance metric and therefore may be subject to early stop termination. The FL service may generate notes for the early stop of a HP configuration that may later be shared with the FL user when reporting the results. Steps 4 to 8 may be repeated for multiple training rounds until all the data of the dataset have been exhausted. The FL service may also expose the real time updates of the training results to the FL user and the FL user may be allowed to perform early stop termination of certain underperforming HP configurations.

[0122] FIG. 11 shows a GUI dashboard the FL service may present to the FL user showing the status of the FL HPO training process. The GUI highlights an important aspect of the FL driven HPO process: dynamic HP reconfigurations are possible to address HP configurations whose performance metric is not converging towards an acceptable performance level. As shown in the figure, the performance metric of HP configuration 4 is lagging the performance metrics of the other HP configurations. At the 35% completion stage, the FL service may decide to perform an early stop of FL HPO training for HP configuration 4 and start two new HP configurations 6 and 7. Due to the federated nature of training, the FL service is able to monitor the FL HPO performance in real time and dynamically adjust the HP configuration if necessary. The stopping criteria may be provided by the FL user, from local or configured policies, or determined by the FL service, e.g., by evaluating and comparing performance metrics of all HP configurations at any instance in time. The new HP configurations may be obtained from HP configurations removed during the FL HP search space reduction method or determined by the FL service and/or FL user.

[0123] As shown in FIG. 9, at step 9, the FL user may make an HP configuration update to change parameters for the remaining FL HPO process. For example, the FL user may notice that a certain HP configuration is not showing convergence towards the target performance metric and the FL user may want to early-stop the training associated with that HP configuration. As another example, the FL user may be satisfied with the performance shown by a few HP configurations and may decide to cancel the remaining FL HPO training rounds. Steps 4 to 9 may be repeated until all scheduled FL HPO training rounds have completed. Similarly, the FL service may also perform HP configuration updates in which the FL service adds/removes HP configurations based on real time updates of the performance metrics of different HP configurations. The FL user may have enabled the FL service to manage the FL HPO training process by selecting the automated training indicator as shown in the table above.

[0124] At step 10, at the completion of all the FL HPO training rounds, the FL service may evaluate the final performance of each HP configurations to determine the best performing ML model or models. Note that each HP configuration (except for those that are early terminated) after all training have been completed will generate a resulting trained model. For cases in which different ML algorithms were specified, the FL service may present the best performing models according to the ML algorithm type, e.g. neural networks, random forests, support vector machines, etc. The FL service may rank the ML models according to the performance metric computed for each HP configuration. [0125] At step 1 1, the FL service may send the final results of the FL HPO training to the FL user and provide a list of the best performing ML models in a notification message sent to the FL user or via a GUI on a dashboard display. Statistics associated with each HP configuration may be provided in the notification message to provide more information of other metrics the FL service may have maintained during the training. For example, the FL service may provide performance metrics, trend lines of the accuracy function through the various training rounds, and the reason for the early stop of certain HP configurations. The FL service may also provide summary information of the FL HPO training such as FL training identifier, FL HPO identifier, a list of the participating FL clients; dataset identifiers with corresponding metrics for size, coverage and distribution percentages, aggregate dataset size, the total number of training rounds; and the total elapse time of the training. The FL service may also provide recommendations on what additional HP configurations may be worthwhile to pursue for future FL HPO runs.

[0126] Even though the above solutions were described in the context of hyperparameter optimization for federated learning, the solutions may also apply to hyperparameter optimization for general machine learning. For example, the dataset subsampling and hyperparameter search space reduction method may be incorporated into general ML hyperparameter optimization as part of energy efficient initiatives or if the HP search space is large and it is desired to reduce the search space to obtain results quicker.

[0127] The aforementioned services may be realized as one or more RESTful APIs made available by a FL service to FL users. The APIs may incorporate the request parameters as previously mentioned to enable FL users the ability to configure the FL HPO process and enable the FL service to perform the necessary actions for managing the FL HPO. FIG. 12 shows an example RESTful resource in which information about the FL HPO process may be stored and referenced.

[0128] The RESTful resource, which is represented as rectangles in FIG. 12, may be created in response to any of the aforementioned methods. A resource may store the informational elements as shown in the tables above to enable the management of FL HPO training by a FL service. A FL Training Identifier resource may be assigned by the FL service to associate FL HPO training to a FL user and parameters such as those found in the table above for a particular FL HP configuration may be saved in the FL HPO Identifier resource. Information about the ML application, such as use case, ML algorithms, and function types, may be stored in the ML Application attribute, which is designated with a rounded rectangle. The FL Training Requirements and Dataset Requirements resources may contain information as shown in the tables above. Finally, a FL Training Status resource may provide real-time status of the FL HPO process for FL users to monitor and modify (if necessary).

[0129] In addition to RESTful APIs that may be offered, FL users may also access the aforementioned services using graphical user interfaces that may be available through a dashboard provided by the FL service. FIG. 13 shows an example of a GUI that may be provided by an FL service to FL users for requesting FL training. The information presented in the GUI may apply to information an FL user provides to the FL service as shown in step 2 of FIG. 6B. As shown by the GUI, the FL User may have the ability to specify an ML application (or use case, what to predict), ML algorithms to use for FL training, the target feature and a minimum number features required for a dataset, a feature list, a minimum number of FL clients, dataset size and data coverage and distribution requirements, and other parameters to configure FL training.

[0130] A FL service may also expose an API and/or GUI for FL users to configure HP configurations for FL training. FIG. 14 shows a GUI the FL service may present to FL users for requesting FL HP configurations. The information presented in the GUI may apply to information an FL user may provide to the FL service as shown in step 6 of FIG. 6B. An FL user may make decisions using the information provided from step 5 of the same method, such as selections of the specified range of HP values or limit the HP range further to reduce the HP search space. The FL user may send the information in a request to the FL service to indicate the selection of FL clients, hyperparameters, and the associated hyperparameter ranges for the FL HPO training. The FL user may also provide an allowed tuning indication that would allow the FL service to derive additional HP configurations from the ones that the FL user had provided.

[0131] The 3rd Generation Partnership Project (3GPP) develops technical standards for cellular telecommunications network technologies, including radio access, the core transport network, and service capabilities - including work on codecs, security, and quality of service. Recent radio access technology (RAT) standards comprise WCDMA (commonly referred as 3G), LTE (commonly referred as 4G), LTE-Advanced standards, and New Radio (NR), which is also referred to as “5G”. 3GPP NR standards development is expected to continue and comprise the definition of next generation radio access technology (new RAT), which is expected to comprise the provision of new flexible radio access below 7 GHz, and the provision of new ultra-mobile broadband radio access above 7 GHz. The flexible radio access is expected to consist of a new, non-backwards compatible radio access in new spectrum below 7 GHz, and it is expected to comprise different operating modes that may be multiplexed together in the same spectrum to address a broad set of 3 GPP NR use cases with diverging requirements. The ultra- mobile broadband is expected to comprise cmWave and mmWave spectrum that may provide the opportunity for ultra-mobile broadband access for, e.g., indoor applications and hotspots. In particular, the ultra-mobile broadband is expected to share a common design framework with the flexible radio access below 7 GHz, with cmWave and mmWave specific design optimizations.

[0132] 3GPP has identified a variety of use cases that NR is expected to support, resulting in a wide variety of user experience requirements for data rate, latency, and mobility. The use cases comprise the following general categories: enhanced mobile broadband (eMBB) ultra-reliable low-latency Communication (URLLC), massive machine type communications (mMTC), network operation (e.g., network slicing, routing, migration and interworking, energy savings), and enhanced vehicle-to-everything (eV2X) communications, which may comprise any of Vehicle-to-Vehicle Communication (V2V), Vehicle-to-Infrastructure Communication (V2I), Vehicle-to-Network Communication (V2N), Vehicle-to-Pedestrian Communication (V2P), and vehicle communications with other entities. Specific service and applications in these categories comprise, e.g., monitoring and sensor networks, device remote controlling, bi-directional remote controlling, personal cloud computing, video streaming, wireless cloud-based office, first responder connectivity, automotive ecall, disaster alerts, real-time gaming, multi-person video calls, autonomous driving, augmented reality, tactile internet, virtual reality, home automation, robotics, and aerial drones to name a few. All of these use cases and others are contemplated herein.

[0133] FIG. 15A illustrates an example communications system 100 in which the systems, methods, and apparatuses described and claimed herein may be used. The communications system 100 may comprise wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, 102e, 102f, and/or 102g, which generally or collectively may be referred to as WTRU 102 or WTRUs 102. One or more of these WTRUs (or in any of FIGs. 15B, 15C, or 15D) may comprise an FL user device or FL client, as discussed above, for example, in connection with FIG. 6A. The communications system 100 may comprise, a radio access network (RAN) 103/104/105/103b/l 04b/105b, a core network 106/107/109, a public switched telephone network (PSTN) 108, the Internet 110, other networks 112, and Network Services 113. 113. Network Services 113 may comprise, for example, a V2X server, V2X functions, a ProSe server, ProSe functions, loT services, video streaming, federated learning (FL) services, and/or edge computing, etc. For example, the FL service described above in connection with FIG. 6A may reside on a server, or other computing device, communicatively coupled to the internet 110, core network 106/107/109, PSTN 108 and/or other networks 112 (or similar network shown in FIGs. 15B, 15C, or 15D).

[0134] It may be appreciated that the concepts disclosed herein may be used with any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102 may be any type of apparatus or device configured to operate and/or communicate in a wireless environment. In the example of FIG. 15 A, each of the WTRUs 102a-d is depicted in FIGs. ISA- ISE as a hand-held wireless communications apparatus. It is understood that with the wide variety of use cases contemplated for wireless communications, each WTRU may comprise or be comprised in any type of apparatus or device configured to transmit and/or receive wireless signals, including, by way of example only, user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a tablet, a netbook, a notebook computer, a personal computer, a wireless sensor, consumer electronics, a wearable device such as a smart watch or smart clothing, a medical or eHealth device, a robot, industrial equipment, a drone, a vehicle such as a car, bus or truck, a train, or an airplane, and the like.

[0135] The communications system 100 may also comprise a base station 114a and a base station 114b. In the example of FIG. 15A, each base stations 114a and 114b is depicted as a single element. In practice, the base stations 114a and 114b may comprise any number of interconnected base stations and/or network elements. Base stations 114a may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, and 102c to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, Network Services 113, and/or the other networks 112. Similarly, base station 114b may be any type of device configured to wiredly and/or wirelessly interface with at least one of the Remote Radio Heads (RRHs) 118a, 118b, Transmission and Reception Points (TRPs) 119a, 1 19b, and/or Roadside Units (RSUs) 120a and 120b to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, other networks 112, and/or Network Services 113. RRHs 118a, 118b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102, e.g., WTRU 102c, to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, Network Services 113, and/or other networks 112.

[0136] TRPs 119a, 119b may be any type of device configured to wirelessly interface with at least one of the WTRU 102d, to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, Network Services 113, and/or other networks 112. RSUs 120a and 120b may be any type of device configured to wirelessly interface with at least one of the WTRU 102e or 102f, to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, other networks 112, and/or Network Services 113. By way of example, the base stations 114a, 114b may be a Base Transceiver Station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a Next Generation Node-B (gNode B), a satellite, a site controller, an access point (AP), a wireless router, and the like.

[0137] The base station 114a may be part of the RAN 103/104/105, which may also comprise other base stations and/or network elements (not shown), such as a Base Station Controller (BSC), a Radio Network Controller (RNC), relay nodes, etc. Similarly, the base station 114b may be part of the RAN 103b/l 04b/l 05b, which may also comprise other base stations and/or network elements (not shown), such as a BSC, a RNC, relay nodes, etc. The base station 114a may be configured to transmit and/or receive wireless signals within a particular geographic region, which may be referred to as a cell (not shown). Similarly, the base station 114b may be configured to transmit and/or receive wired and/or wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, for example, the base station 114a may comprise three transceivers, e.g., one for each sector of the cell. The base station 114a may employ Multiple- Input Multiple Output (MIMO) technology and, therefore, may utilize multiple transceivers for each sector of the cell, for instance. [0138] The base station 1 14a may communicate with one or more of the WTRUs 102a, 102b, 102c, and 102g over an air interface 115/116/117, which may be any suitable wireless communication link (e.g., Radio Frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, cmWave, mmWave, etc.). The air interface 115/116/117 may be established using any suitable Radio Access Technology (RAT).

[0139] The base station 114b may communicate with one or more of the RRHs 118a and 118b, TRPs 119a and 119b, and/or RSUs 120a and 120b, over a wired or air interface 115b/l 16b/l 17b, which may be any suitable wired (e.g., cable, optical fiber, etc.) or wireless communication link (e.g., RF, microwave, IR, UV, visible light, cmWave, mmWave, etc.). The air interface 115b/l 16b/l 17b may be established using any suitable RAT.

[0140] The RRHs 118a, 118b, TRPs 119a, 119b and/or RSUs 120a, 120b, may communicate with one or more of the WTRUs 102c, 102d, 102e, 102f over an air interface 115c/l 16c/l 17c, which may be any suitable wireless communication link (e.g., RF, microwave, IR, ultraviolet UV, visible light, cmWave, mmWave, etc.) The air interface 115c/l 16c/l 17c may be established using any suitable RAT.

[0141] The WTRUs 102 may communicate with one another over a direct air interface 115d/l 16d/l 17d, such as Sidelink communication which may be any suitable wireless communication link (e.g., RF, microwave, IR, ultraviolet UV, visible light, cmWave, mmWave, etc.) The air interface 115d/l 16d/l 17d may be established using any suitable RAT.

[0142] The communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC- FDMA, and the like. For example, the base station 114a in the RAN 103/104/105 and the WTRUs 102a, 102b, 102c, or RRHs 118a, 118b, TRPs 119a, 119b and/or RSUs 120a and 120b in the RAN 103b/l 04b/l 05b and the WTRUs 102c, 102d, 102e, and 102f, may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115/116/117 and/or 115c/l 16c/l 17c respectively using Wideband CDMA (WCDMA). WCDMA may comprise communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may comprise High-Speed Downlink Packet Access (HSDPA) and/or High-Speed Uplink Packet Access (HSUPA). [0143] The base station 1 14a in the RAN 103/104/105 and the WTRUs 102a, 102b, 102c, and 102g, or RRHs 118a and 118b, TRPs 119a and 119b, and/or RSUs 120a and 120b in the RAN 103b/ 104b/l 05b and the WTRUs 102c, 102d, may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 115/116/117 or 115c/l 16c/l 17c respectively using Long Term Evolution (LTE) and/or LTE- Advanced (LTE-A), for example. The air interface 115/116/117 or 115c/l 16c/l 17c may implement 3GPP NR technology. The LTE and LTE-A technology may comprise LTE D2D and/or V2X technologies and interfaces (such as Sidelink communications, etc.) Similarly, the 3GPP NR technology may comprise NR V2X technologies and interfaces (such as Sidelink communications, etc.)

[0144] The base station 114a in the RAN 103/104/105 and the WTRUs 102a, 102b, 102c, and 102g or RRHs 118a and 118b, TRPs 119a and 119b, and/or RSUs 120a and 120b in the RAN 103b/l 04b/l 05b and the WTRUs 102c, 102d, 102e, and 102f may implement radio technologies such as IEEE 802.16 (e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Interim Standard 2000 (IS- 2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0145] The base station 114c in FIG. 15A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a train, an aerial, a satellite, a manufactory, a campus, and the like. The base station 114c and the WTRUs 102, e.g., WTRU 102e, may implement a radio technology such as IEEE 802.11 to establish a Wireless Local Area Network (WLAN). Similarly, the base station 114c and the WTRUs 102, e.g., WTRU 102d, may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). The base station 114c and the WTRUs 102, e.g., WRTU 102e, may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, NR, etc.) to establish a picocell or femtocell. As shown in FIG. 15A, the base station 114c may have a direct connection to the Internet 110. Thus, the base station 114c may not be required to access the Internet 110 via the core network 106/107/109. [0146] The RAN 103/104/105 and/or RAN 103b/l 04b/l 05b may be in communication with the core network 106/107/109, which may be any type of network configured to provide voice, data, messaging, authorization and authentication, applications, and/or Voice Over Internet Protocol (VoIP) services to one or more of the WTRUs 102. For example, the core network 106/107/109 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, packet data network connectivity, Ethernet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication.

[0147] Although not shown in FIG. 15 A, it may be appreciated that the RAN 103/104/105 and/or RAN 103b/l 04b/l 05b and/or the core network 106/107/109 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 103/104/105 and/or RAN 103b/l 04b/ 105b or a different RAT. For example, in addition to being connected to the RAN 103/104/105 and/or RAN 103b/l 04b/l 05b, which may be utilizing an E-UTRA radio technology, the core network 106/107/109 may also be in communication with another RAN (not shown) employing a GSM or NR radio technology.

[0148] The core network 106/107/109 may also serve as a gateway for the WTRUs 102 to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may comprise circuit-switched telephone networks that provide Plain Old Telephone Service (POTS). The Internet 110 may comprise a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and the internet protocol (IP) in the TCP/IP internet protocol suite. The other networks 112 may comprise wired or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may comprise any type of packet data network (e.g., an IEEE 802.3 Ethernet network) or another core network connected to one or more RANs, which may employ the same RAT as the RAN 103/104/105 and/or RAN 103b/l 04b/l 05b or a different RAT.

[0149] Some or all of the WTRUs 102a, 102b, 102c, 102d, 102e, and 102f in the communications system 100 may comprise multi-mode capabilities, e.g., the WTRUs 102a, 102b, 102c, 102d, 102e, and 102f may comprise multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102g shown in FIG. 15A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114c, which may employ an IEEE 802 radio technology.

[0150] Although not shown in FIG. 15 A, it may be appreciated that a User Equipment may make a wired connection to a gateway. The gateway maybe a Residential Gateway (RG). The RG may provide connectivity to a Core Network 106/107/109. It may be appreciated that many of the ideas contained herein may equally apply to UEs that are WTRUs and UEs that use a wired connection to connect to a network. For example, the ideas that apply to the wireless interfaces 115, 116, 117 and 115c/l 16c/l 17c may equally apply to a wired connection.

[0151] FIG. 15B is a system diagram of an example RAN 103 and core network 106. As noted above, the RAN 103 may employ a UTRA radio technology to communicate with the WTRUs 102a, 102b, and 102c over the air interface 115. The RAN 103 may also be in communication with the core network 106. As shown in FIG. 15B, the RAN 103 may comprise Node-Bs 140a, 140b, and 140c, which may each comprise one or more transceivers for communicating with the WTRUs 102a, 102b, and 102c over the air interface 115. The Node-Bs 140a, 140b, and 140c may each be associated with a particular cell (not shown) within the RAN 103. The RAN 103 may also comprise RNCs 142a, 142b. It may be appreciated that the RAN 103 may comprise any number of Node-Bs and Radio Network Controllers (RNCs.)

[0152] As shown in FIG. 15B, the Node-Bs 140a, 140b may be in communication with the RNC 142a. Additionally, the Node-B 140c may be in communication with the RNC 142b. The Node-Bs 140a, 140b, and 140c may communicate with the respective RNCs 142a and 142b via an lub interface. The RNCs 142a and 142b may be in communication with one another via an lur interface. Each of the RNCs 142aand 142b may be configured to control the respective Node- Bs 140a, 140b, and 140c to which it is connected. In addition, each of the RNCs 142aand 142b may be configured to carry out or support other functionality, such as outer loop power control, load control, admission control, packet scheduling, handover control, macro-diversity, security functions, data encryption, and the like.

[0153] The core network 106 shown in FIG. 15B may comprise a media gateway (MGW) 144, a Mobile Switching Center (MSC) 146, a Serving GPRS Support Node (SGSN) 148, and/or a Gateway GPRS Support Node (GGSN) 150. While each of the foregoing elements are depicted as part of the core network 106, it may be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator. [0154] The RNC 142a in the RAN 103 may be connected to the MSC 146 in the core network 106 via an luCS interface. The MSC 146 may be connected to the MGW 144. The MSC 146 and the MGW 144 may provide the WTRUs 102a, 102b, and 102c with access to circuit- switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, and 102c, and traditional land-line communications devices.

[0155] The RNC 142a in the RAN 103 may also be connected to the SGSN 148 in the core network 106 via an luPS interface. The SGSN 148 may be connected to the GGSN 150. The SGSN 148 and the GGSN 150 may provide the WTRUs 102a, 102b, and 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between and the WTRUs 102a, 102b, and 102c, and IP-enabled devices.

[0156] The core network 106 may also be connected to the other networks 112, which may comprise other wired or wireless networks that are owned and/or operated by other service providers.

[0157] FIG. 15C is a system diagram of an example RAN 104 and core network 107. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, and 102c over the air interface 116. The RAN 104 may also be in communication with the core network 107.

[0158] The RAN 104 may comprise eNode-Bs 160a, 160b, and 160c, though it may be appreciated that the RAN 104 may comprise any number of eNode-Bs. The eNode-Bs 160a, 160b, and 160c may each comprise one or more transceivers for communicating with the WTRUs 102a, 102b, and 102c over the air interface 116. For example, the eNode-Bs 160a, 160b, and 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a.

[0159] Each of the eNode-Bs 160a, 160b, and 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the uplink and/or downlink, and the like. As shown in FIG. 15C, the eNode-Bs 160a, 160b, and 160c may communicate with one another over an X2 interface.

[0160] The core network 107 shown in FIG. 15C may comprise a Mobility

Management Gateway (MME) 162, a serving gateway 164, and a Packet Data Network (PDN) gateway 166. While each of the foregoing elements are depicted as part of the core network 107, it may be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[0161] The MME 162 may be connected to each of the eNode-Bs 160a, 160b, and 160c in the RAN 104 via an SI interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, and 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, and 102c, and the like. The MME 162 may also provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM or WCDMA.

[0162] The serving gateway 164 may be connected to each of the eNode-Bs 160a, 160b, and 160c in the RAN 104 via the SI interface. The serving gateway 164 may generally route and forward user data packets to/from the WTRUs 102a, 102b, and 102c. The serving gateway 164 may also perform other functions, such as anchoring user planes during inter-eNode B handovers, triggering paging when downlink data is available for the WTRUs 102a, 102b, and 102c, managing and storing contexts of the WTRUs 102a, 102b, and 102c, and the like.

[0163] The serving gateway 164 may also be connected to the PDN gateway 166, which may provide the WTRUs 102a, 102b, and 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c, and IP-enabled devices.

[0164] The core network 107 may facilitate communications with other networks. For example, the core network 107 may provide the WTRUs 102a, 102b, and 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, and 102c and traditional land-line communications devices. For example, the core network 107 may comprise, or may communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between the core network 107 and the PSTN 108. In addition, the core network 107 may provide the WTRUs 102a, 102b, and 102c with access to the networks 112, which may comprise other wired or wireless networks that are owned and/or operated by other service providers.

[0165] FIG. 15D is a system diagram of an example RAN 105 and core network 109. The RAN 105 may employ an NR radio technology to communicate with the WTRUs 102a and 102b over the air interface 117. The RAN 105 may also be in communication with the core network 109. A Non-3GPP Interworking Function (N3IWF) 199 may employ a non-3GPP radio technology to communicate with the WTRU 102c over the air interface 198. The N3IWF 199 may also be in communication with the core network 109.

[0166] The RAN 105 may comprise gNode-Bs 180a and 180b. It may be appreciated that the RAN 105 may comprise any number of gNode-Bs. The gNode-Bs 180a and 180b may each comprise one or more transceivers for communicating with the WTRUs 102a and 102b over the air interface 117. When integrated access and backhaul connection are used, the same air interface may be used between the WTRUs and gNode-Bs, which may be the core network 109 via one or multiple gNBs. The gNode-Bs 180a and 180b may implement MIMO, MU-MIMO, and/or digital beamforming technology. Thus, the gNode-B 180a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a. It should be appreciated that the RAN 105 may employ of other types of base stations such as an eNode-B. It may also be appreciated the RAN 105 may employ more than one type of base station. For example, the RAN may employ eNode-Bs and gNode-Bs.

[0167] The N3IWF 199 may comprise a non-3GPP Access Point 180c. It may be appreciated that the N3IWF 199 may comprise any number of non-3GPP Access Points. The non-3GPP Access Point 180c may comprise one or more transceivers for communicating with the WTRUs 102c over the air interface 198. The non-3GPP Access Point 180c may use the 802.11 protocol to communicate with the WTRU 102c over the air interface 198.

[0168] Each of the gNode-Bs 180a and 180b may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the uplink and/or downlink, and the like. As shown in FIG. 15D, the gNode-Bs 180a and 180b may communicate with one another over an Xn interface, for example.

[0169] The core network 109 shown in FIG. 15D may be a 5G core network (5GC). The core network 109 may offer numerous communication services to customers who are interconnected by the radio access network. The core network 109 comprises a number of entities that perform the functionality of the core network. As used herein, the term “core network entity” or “network function” refers to any entity that performs one or more functionalities of a core network. It is understood that such core network entities may be logical entities that are implemented in the form of computer-executable instructions (software) stored in a memory of, and executing on a processor of, an apparatus configured for wireless and/or network communications or a computer system, such as system 90 illustrated in FIG. 15G.

[0170] In the example of FIG. 15D, the 5G Core Network 109 may comprise an access and mobility management function (AMF) 172, a Session Management Function (SMF) 174, User Plane Functions (UPFs) 176a and 176b, a User Data Management Function (UDM) 197, an Authentication Server Function (AUSF) 190, a Network Exposure Function (NEF) 196, a Policy Control Function (PCF) 184, a Non-3GPP Interworking Function (N3IWF) 199, a User Data Repository (UDR) 178. While each of the foregoing elements are depicted as part of the 5G core network 109, it may be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator. It may also be appreciated that a 5G core network may not consist of all of these elements, may consist of additional elements, and may consist of multiple instances of each of these elements. FIG. 15D shows that network functions directly connect to one another, however, it should be appreciated that they may communicate via routing agents such as a diameter routing agent or message buses.

[0171] In the example of FIG. 15D, connectivity between network functions is achieved via a set of interfaces, or reference points. It may be appreciated that network functions may be modeled, described, or implemented as a set of services that are invoked, or called, by other network functions or services. Invocation of a Network Function service may be achieved via a direct connection between network functions, an exchange of messaging on a message bus, calling a software function, etc.

[0172] The AMF 172 may be connected to the RAN 105 via an N2 interface and may serve as a control node. For example, the AMF 172 may be responsible for registration management, connection management, reachability management, access authentication, access authorization. The AMF may be responsible forwarding user plane tunnel configuration information to the RAN 105 via the N2 interface. The AMF 172 may receive the user plane tunnel configuration information from the SMF via an N11 interface. The AMF 172 may generally route and forward NAS packets to/from the WTRUs 102a, 102b, and 102c via an N1 interface. The N1 interface is not shown in FIG. 15D.

[0173] The SMF 174 may be connected to the AMF 172 via an N11 interface. Similarly the SMF may be connected to the PCF 184 via an N7 interface, and to the UPFs 176a and 176b via an N4 interface. The SMF 174 may serve as a control node. For example, the SMF 174 may be responsible for Session Management, IP address allocation for the WTRUs 102a, 102b, and 102c, management and configuration of traffic steering rules in the UPF 176a and UPF 176b, and generation of downlink data notifications to the AMF 172.

[0174] The UPF 176a and UPF176b may provide the WTRUs 102a, 102b, and 102c with access to a Packet Data Network (PDN), such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, and 102c and other devices. The UPF 176a and UPF 176b may also provide the WTRUs 102a, 102b, and 102c with access to other types of packet data networks. For example, Other Networks 112 may be Ethernet Networks or any type of network that exchanges packets of data. The UPF 176a and UPF 176b may receive traffic steering rules from the SMF 174 via the N4 interface. The UPF 176a and UPF 176b may provide access to a packet data network by connecting a packet data network with an N6 interface or by connecting to each other and to other UPFs via an N9 interface. In addition to providing access to packet data networks, the UPF 176 may be responsible packet routing and forwarding, policy rule enforcement, quality of service handling for user plane traffic, downlink packet buffering.

[0175] The AMF 172 may also be connected to the N3IWF 199, for example, via an N2 interface. The N3IWF facilitates a connection between the WTRU 102c and the 5G core network 170, for example, via radio interface technologies that are not defined by 3GPP. The AMF may interact with the N3IWF 199 in the same, or similar, manner that it interacts with the RAN 105.

[0176] The PCF 184 may be connected to the SMF 174 via an N7 interface, connected to the AMF 172 via an N15 interface, and to an Application Function (AF) 188 via an N5 interface. The N15 and N5 interfaces are not shown in FIG. 15D. The PCF 184 may provide policy rules to control plane nodes such as the AMF 172 and SMF 174, allowing the control plane nodes to enforce these rules. The PCF 184, may send policies to the AMF 172 for the WTRUs 102a, 102b, and 102c so that the AMF may deliver the policies to the WTRUs 102a, 102b, and 102c via an N1 interface. Policies may then be enforced, or applied, at the WTRUs 102a, 102b, and 102c.

[0177] The UDR 178 may act as a repository for authentication credentials and subscription information. The UDR may connect to network functions, so that network function may add to, read from, and modify the data that is in the repository. For example, the UDR 178 may connect to the PCF 184 via an N36 interface. Similarly, the UDR 178 may connect to the NEF 196 via an N37 interface, and the UDR 178 may connect to the UDM 197 via an N35 interface.

[0178] The UDM 197 may serve as an interface between the UDR 178 and other network functions. The UDM 197 may authorize network functions to access of the UDR 178. For example, the UDM 197 may connect to the AMF 172 via an N8 interface, the UDM 197 may connect to the SMF 174 via an N10 interface. Similarly, the UDM 197 may connect to the AUSF 190 via an N13 interface. The UDR 178 and UDM 197 may be tightly integrated.

[0179] The AUSF 190 performs authentication related operations and connects to the UDM 178 via an N13 interface and to the AMF 172 via an N12 interface.

[0180] The NEF 196 exposes capabilities and services in the 5G core network 109 to Application Functions (AF) 188. Exposure may occur on the N33 API interface. The NEF may connect to an AF 188 via an N33 interface and it may connect to other network functions in order to expose the capabilities and services of the 5G core network 109.

[0181] Application Functions 188 may interact with network functions in the 5G Core Network 109. Interaction between the Application Functions 188 and network functions may be via a direct interface or may occur via the NEF 196. The Application Functions 188 may be considered part of the 5G Core Network 109 or may be external to the 5G Core Network 109 and deployed by enterprises that have a business relationship with the mobile network operator.

[0182] Network Slicing is a mechanism that may be used by mobile network operators to support one or more ‘virtual’ core networks behind the operator’s air interface. This involves ‘slicing’ the core network into one or more virtual networks to support different RANs or different service types running across a single RAN. Network slicing enables the operator to create networks customized to provide optimized solutions for different market scenarios which demands diverse requirements, e.g., in the areas of functionality, performance and isolation.

[0183] 3GPP has designed the 5G core network to support Network Slicing. Network Slicing is a good tool that network operators may use to support the diverse set of 5G use cases (e g., massive loT, critical communications, V2X, and enhanced mobile broadband) which demand very diverse and sometimes extreme requirements. Without the use of network slicing techniques, it is likely that the network architecture would not be flexible and scalable enough to efficiently support a wider range of use cases need when each use case has its own specific set of performance, scalability, and availability requirements. Furthermore, introduction of new network services should be made more efficient.

[0184] Referring again to FIG. 15D, in a network slicing scenario, a WTRU 102a, 102b, or 102c may connect to an AMF 172, via an N1 interface. The AMF may be logically part of one or more slices. The AMF may coordinate the connection or communication of WTRU 102a, 102b, or 102c with one or more UPF 176a and 176b, SMF 174, and other network functions. Each of the UPFs 176a and 176b, SMF 174, and other network functions may be part of the same slice or different slices. When they are part of different slices, they may be isolated from each other in the sense that they may utilize different computing resources, security credentials, etc.

[0185] The core network 109 may facilitate communications with other networks. For example, the core network 109 may comprise, or may communicate with, an IP gateway, such as an IP Multimedia Subsystem (IMS) server, that serves as an interface between the 5G core network 109 and a PSTN 108. For example, the core network 109 may comprise, or communicate with a short message service (SMS) service center that facilities communication via the short message service. For example, the 5G core network 109 may facilitate the exchange of non-IP data packets between the WTRUs 102a, 102b, and 102c and servers or applications functions 188. In addition, the core network 170 may provide the WTRUs 102a, 102b, and 102c with access to the networks 112, which may comprise other wired or wireless networks that are owned and/or operated by other service providers.

[0186] The core network entities described herein and illustrated in FIGs. 15 A, 15C, 15D, and 15E are identified by the names given to those entities in certain existing 3GPP specifications, but it is understood that in the future those entities and functionalities may be identified by other names and certain entities or functions may be combined in future specifications published by 3GPP, including future 3GPP NR specifications. Thus, the particular network entities and functionalities described and illustrated in FIGs. 15 A, 15B, 15C, 15D, and 15E are provided by way of example only, and it is understood that the subject matter disclosed and claimed herein may be embodied or implemented in any similar communication system, whether presently defined or defined in the future.

[0187] FIG. 15E illustrates an example communications system 111 in which the systems, methods, apparatuses described herein may be used. Communications system 111 may comprise Wireless Transmit/Receive Units (WTRUs) A, B, C, D, E, F, a base station gNB 121, a V2X server 124, and Road Side Units (RSUs) 123a and 123b. In practice, the concepts presented herein may be applied to any number of WTRUs, base station gNBs, V2X networks, and/or other network elements. One or several or all WTRUs A, B, C, D, E, and F may be out of range of the access network coverage 131. WTRUs A, B, and C form a V2X group, among which WTRU A is the group lead and WTRUs B and C are group members.

[0188] WTRUs A, B, C, D, E, and F may communicate with each other over a Uu interface 129 via the gNB 121 if they are within the access network coverage 131. In the example of FIG. 15E, WTRUs B and F are shown within access network coverage 131. WTRUs A, B, C, D, E, and F may communicate with each other directly via a Sidelink interface (e.g., PC5 or NR PC5) such as interface 125a, 125b, or 128, whether they are under the access network coverage 131 or out of the access network coverage 131. For instance, in the example of FIG. 15E, WRTU D, which is outside of the access network coverage 131, communicates with WTRU F, which is inside the coverage 131.

[0189] WTRUs A, B, C, D, E, and F may communicate with RSU 123a or 123b via a Vehicle-to-Network (V2N) 133 or Sidelink interface 125b. WTRUs A, B, C, D, E, and F may communicate to a V2X Server 124 via a Vehicle-to-Infrastructure (V2I) interface 127. WTRUs A, B, C, D, E, and F may communicate to another UE via a Vehicle-to-Person (V2P) interface 128.

[0190] FIG. 15F is a block diagram of an example apparatus or device WTRU 102 that may be configured for wireless communications and operations in accordance with the systems, methods, and apparatuses described herein, such as a WTRU 102 of FIG. 15A, 15B, 15C, 15D, or 15E. For example, the WTRU 102 of FIG. 15F may be used to implement an FL user device or FL client, as described above. As shown in FIG. 15F, the example WTRU 102 may comprise a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad/indicators 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It may be appreciated that the WTRU 102 may comprise any sub-combination of the foregoing elements. Also, the base stations 114a and 114b, and/or the nodes that base stations 114a and 114b may represent, such as but not limited to transceiver station (BTS), a Node-B, a site controller, an access point (AP), a home node-B, an evolved home node-B (eNodeB), a home evolved node-B (HeNB), a home evolved node-B gateway, a next generation node-B (gNode-B), and proxy nodes, among others, may comprise some or all of the elements depicted in FIG. 15F and described herein.

[0191] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 15F depicts the processor 118 and the transceiver 120 as separate components, it may be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0192] The transmit/receive element 122 of a UE may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a of FIG. 15A) over the air interface 115/116/117 or another UE over the air interface 115d/l 16d/l 17d. For example, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. The transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. The transmit/receive element 122 may be configured to transmit and receive both RF and light signals. It may be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless or wired signals.

[0193] In addition, although the transmit/receive element 122 is depicted in FIG. 15F as a single element, the WTRU 102 may comprise any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, the WTRU 102 may comprise two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 115/116/117.

[0194] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may comprise multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, for example NR and IEEE 802.11 or NR and E- UTRA, or to communicate with the same RAT via multiple beams to different RRHs, TRPs, RSUs, or nodes.

[0195] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/mi crophone 124, the keypad 126, and/or the display/touchpad/indicators 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit. The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad/indicators 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The nonremovable memory 130 may comprise random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may comprise a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. The processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server that is hosted in the cloud or in an edge computing platform or in a home computer (not shown).

[0196] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may comprise one or more dry cell batteries, solar cells, fuel cells, and the like.

[0197] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 115/116/117 from a base station (e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It may be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method. [0198] The processor 118 may further be coupled to other peripherals 138, which may comprise one or more software and/or hardware modules that provide additional features, functionality, and/or wired or wireless connectivity. For example, the peripherals 138 may comprise various sensors such as an accelerometer, biometrics (e.g., finger print) sensors, an e- compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port or other interconnect interfaces, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[0199] The WTRU 102 may be comprised in other apparatuses or devices, such as a sensor, consumer electronics, a wearable device such as a smart watch or smart clothing, a medical or eHealth device, a robot, industrial equipment, a drone, a vehicle such as a car, truck, train, or an airplane. The WTRU 102 may connect to other components, modules, or systems of such apparatuses or devices via one or more interconnect interfaces, such as an interconnect interface that may comprise one of the peripherals 138.

[0200] FIG. 15G is a block diagram of an exemplary computing system 90 in which one or more apparatuses of the communications networks illustrated in FIGs. 15 A, 15C, 15D and 15E may be embodied, such as certain nodes or functional entities in the RAN 103/104/105, Core Network 106/107/109, PSTN 108, Internet 110, Other Networks 112, or Network Services 113. For example, an FL user device, FL client, or FL service may be embodied in the form of such exemplary computing system 90. Computing system 90 may comprise a computer or server and may be controlled primarily by computer readable instructions, which may be in the form of software, wherever, or by whatever means such software is stored or accessed. Such computer readable instructions may be executed within a processor 91, to cause computing system 90 to do work. The processor 91 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 91 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the computing system 90 to operate in a communications network. Coprocessor 81 is an optional processor, distinct from main processor 91, that may perform additional functions or assist processor 91 . Processor 91 and/or coprocessor 81 may receive, generate, and process data related to the methods and apparatuses disclosed herein.

[0201] In operation, processor 91 fetches, decodes, and executes instructions, and transfers information to and from other resources via the computing system’s main data-transfer path, system bus 80. Such a system bus connects the components in computing system 90 and defines the medium for data exchange. System bus 80 typically comprises data lines for sending data, address lines for sending addresses, and control lines for sending interrupts and for operating the system bus. An example of such a system bus 80 is the PCI (Peripheral Component Interconnect) bus.

[0202] Memories coupled to system bus 80 comprise random access memory (RAM) 82 and read only memory (ROM) 93. Such memories comprise circuitry that allows information to be stored and retrieved. ROMs 93 generally contain stored data that may not easily be modified. Data stored in RAM 82 may be read or changed by processor 91 or other hardware devices. Access to RAM 82 and/or ROM 93 may be controlled by memory controller 92. Memory controller 92 may provide an address translation function that translates virtual addresses into physical addresses as instructions are executed. Memory controller 92 may also provide a memory protection function that isolates processes within the system and isolates system processes from user processes. Thus, a program running in a first mode may access only memory mapped by its own process virtual address space; it may not access memory within another process’s virtual address space unless memory sharing between the processes has been set up.

[0203] In addition, computing system 90 may contain peripherals controller 83 responsible for communicating instructions from processor 91 to peripherals, such as printer 94, keyboard 84, mouse 95, and disk drive 85.

[0204] Display 86, which is controlled by display controller 96, is used to display visual output generated by computing system 90. Such visual output may comprise text, graphics, animated graphics, and video. The visual output may be provided in the form of a graphical user interface (GUI). Display 86 may be implemented with a CRT -based video display, an LCD-based flat-panel display, gas plasma-based flat-panel display, or a touch-panel. Display controller 96 comprises electronic components required to generate a video signal that is sent to display 86. [0205] Further, computing system 90 may contain communication circuitry, such as for example a wireless or wired network adapter 97, that may be used to connect computing system 90 to an external communications network or devices, such as the RAN 103/104/105, Core Network 106/107/109, PSTN 108, Internet 110, WTRUs 102, or Other Networks 112 ofFIGs. 15A, 15B, 15C, 15D, and 15E, to enable the computing system 90 to communicate with other nodes or functional entities of those networks. The communication circuitry, alone or in combination with the processor 91, may be used to perform the transmitting and receiving steps of certain apparatuses, nodes, or functional entities described herein.

[0206] It is understood that any or all of the apparatuses, systems, methods and processes described herein may be embodied in the form of computer executable instructions (e.g., program code) stored on a computer-readable storage medium which instructions, when executed by one or more processors, such as processors 118 or 91, cause the processor(s) to perform and/or implement the systems, methods and processes described herein. Specifically, any of the steps, operations, or functions described herein may be implemented in the form of such computer executable instructions, executing on the processor(s) of an apparatus or computing system configured for wireless and/or wired network communications. Computer readable storage media comprises volatile and nonvolatile, removable and non-removable media implemented in any non-transitory (e.g., tangible or physical) method or technology for storage of information, but such computer readable storage media do not comprise signals. Computer readable storage media comprise, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible or physical medium which may be used to store the desired information and which may be accessed by a computing system.

Claims

CLAIMS What is claimed is:

1. A method for federated learning (FL) hyperparameter (HP) search space reduction, comprising: receiving, by a FL service configured to communicate via a network, from a FL user device that has requested HP optimization in connection with a FL training of a machine learning model, a request to reduce a HP search space, wherein the request to reduce the HP search space comprises an identifier associated with the FL training and one or more requirements associated with reduction of the HP search space; sending, to each of a plurality of FL clients, a request to partition a dataset of the FL client to create a subsampling of the dataset of the FL client for use in HP search space reduction; creating a configuration for HP search space reduction training by the plurality of FL clients based on the one or more requirements received in the request to reduce the HP search space; causing the HP search space reduction training to be performed by the plurality of FL clients based on the configuration; receiving, from each of the plurality of FL clients, a result of the HP search space reduction training performed by the FL client, wherein the result indicates a performance associated with one or more HP configurations; and sending, to the FL user device, information indicative of the results of the HP search space reduction training performed by the plurality of FL clients.

2. The method recited in claim 1, wherein the one or more requirements associated with reduction of the HP search space comprise one or more of: an indication of a target percentage reduction in the HP search space; an indication of a maximum number of HP configurations for the HP search space; an indication of a target percentage for minimizing a loss function or maximizing accuracy associated with the HP optimization; an indication of one or more aggregation strategies for use during the HP search space reduction; an indication of a minimum number of FL clients to be employed for HP search space reduction; an indication of a minimum number of training rounds required for HP search space reduction; an indication of a size of a data partition of a dataset to be used for HP search space reduction; or an indication of minimum data coverage or distribution percentage for a particular feature in a data partition of a dataset to be used for HP search space reduction.

3. The method recited in claim 1, wherein each request to partition a dataset of one of the plurality of FL clients comprises a dataset identifier, an indication of a minimum size of a partition of the dataset of the FL client, and an indication of a minimum data coverage and distribution.

4. The method recited in claim 1, further comprising: creating a RESTful resource for storing at least (i) the one or more requirements received in the request from the FL user device for HP search space reduction, and (ii) information associated with the partitioned datasets of each of the plurality of FL clients; assigning an identifier to the RESTful resource; and sending, to the FL user device, the identifier of the RESTful resource.

5. The method recited in claim 1, wherein the configuration for HP search space reduction training comprises one or more of: a schedule for the HP search space reduction training by the plurality of FL clients; identifiers associated with one or more rounds of training; information associated with each round of training; identifiers of the partitioned datasets of each FL client; parameters associated with the machine learning model; information associated with one or more HP configurations; or an expiration time associated with the training.

6. The method recited in claim 1, further comprising: determining, based upon the results of the HP search space reduction training performed by each of the FL clients, a ranking of the performance of each of the HP configurations; and reducing a number of the one or more HP configurations based on the one or more requirements of the request received from the FL user device.

7. The method of claim 1, wherein the information indicative of the results of the HP search space reduction training performed by the plurality of FL clients is sent to the FL user device via one or more of: a notification message; or a graphical user interface.

8. The method of claim 1, wherein the information indicative of the results of the HP search space reduction training performed by the plurality of FL clients that is sent to the FL user device comprises one or more of: the identifier associated with the FL training of the machine learning model; a list of the best performing HP configurations resulting from the HP search space reduction training; statistics associated with each HP configuration; or one or more recommendations of HP configurations to pursue in connection with FL HP optimization.

9. An apparatus implementing a federated learning (FL) service, the apparatus comprising: one or more processors; and memory storing instruction that, when executed by the one or more processors, cause the server to perform operations comprising: receiving, by the FL service via a network, from a FL user device that has requested hyperparameter (HP) optimization in connection with a FL training of a machine learning model, a request to reduce a HP search space, wherein the request to reduce the HP search space comprises an identifier associated with the FL training and one or more requirements associated with reduction of the HP search space; sending, to each of a plurality of FL clients, a request to partition a dataset of the FL client to create a subsampling of the dataset of the FL client for use in HP search space reduction; creating a configuration for HP search space reduction training by the plurality of FL clients based on the one or more requirements received in the request to reduce the HP search space; causing the HP search space reduction training to be performed by the plurality of FL clients based on the configuration; receiving, from each of the plurality of FL clients, a result of the HP search space reduction training performed by the FL client, wherein the result indicates a performance associated with one or more HP configurations; and sending, to the FL user device, information indicative of the results of the HP search space reduction training performed by the plurality of FL clients.

10. The apparatus recited in claim 9, wherein the one or more requirements associated with reduction of the HP search space comprise one or more of an indication of a target percentage reduction in the HP search space; an indication of a maximum number of HP configurations for the HP search space; an indication of a target percentage for minimizing a loss function or maximizing accuracy associated with the HP optimization; an indication of one or more aggregation strategies for use during the HP search space reduction; an indication of a minimum number of FL clients to be employed for HP search space reduction; an indication of a minimum number of training rounds required for HP search space reduction; an indication of a size of a data partition of a dataset to be used for HP search space reduction; or an indication of minimum data coverage or distribution percentage for a particular feature in a data partition of a dataset to be used for HP search space reduction.

11. The apparatus recited in claim 9, wherein each request to partition a dataset of one of the plurality of FL clients comprises a dataset identifier, an indication of a minimum size of a partition of the dataset of the FL client, and an indication of a minimum data coverage and distribution.

12. The apparatus recited in claim 9, wherein the instructions, when executed by the one or more processors, further cause the server to: create a RESTful resource for storing at least (i) the one or more requirements received in the request from the FL user device for HP search space reduction, and (ii) information associated with the partitioned datasets of each of the plurality of FL clients; assign an identifier to the RESTful resource; and send, to the FL user device, the identifier of the RESTful resource.

13. The apparatus recited in claim 9, wherein the configuration for HP search space reduction training comprises one or more of: a schedule for the HP search space reduction training by the plurality of FL clients; identifiers associated with one or more rounds of training; information associated with each round of training; identifiers of the partitioned datasets of each FL client; parameters associated with the machine learning model; information associated with one or more HP configurations; or an expiration time associated with the training.

14. The apparatus recited in claim 9, wherein the instructions, when executed by the one or more processors, further cause the server to: determine, based upon the results of the HP search space reduction training performed by each of the FL clients, a ranking of the performance of each of the HP configurations; and reduce a number of the one or more HP configurations based on the one or more requirements of the request received from the FL user device.

15. The apparatus of claim 9, wherein the information indicative of the results of the HP search space reduction training performed by the plurality of FL clients is sent to the FL user device via one or more of: a notification message; or a graphical user interface.

16. The apparatus of claim 9, wherein the information indicative of the results of the HP search space reduction training performed by the plurality of FL clients that is sent to the FL user device comprises one or more of: the identifier associated with the FL training of the machine learning model; a list of the best performing HP configurations resulting from the HP search space reduction training; statistics associated with each HP configuration; or one or more recommendations of HP configurations to pursue in connection with FL HP optimization.

17. A method for federated learning (FL) hyperparameter (HP) search space reduction, comprising: sending, by a FL user device that has requested HP optimization in connection with a FL training of a machine learning model, to a FL service configured to communicate via a network with a plurality of FL clients, a request to reduce a HP search space, wherein the request to reduce the HP search space comprises an identifier associated with the FL training and one or more requirements associated with reduction of the HP search space, and wherein the FL service is configure to cause HP search space reduction training to be performed by the plurality of FL clients based on a configuration created based on the one or more requirements of the request; and receiving, by the FL user device from the FL service, information indicative of the results of the HP search space reduction training performed by the plurality of FL clients.

18. The method recited in claim 17, wherein the one or more requirements associated with reduction of the HP search space comprise one or more of: an indication of a target percentage reduction in the HP search space; an indication of a maximum number of HP configurations for the HP search space; an indication of a target percentage for minimizing a loss function or maximizing accuracy associated with the HP optimization; an indication of one or more aggregation strategies for use during the HP search space reduction; an indication of a minimum number of FL clients to be employed for HP search space reduction; an indication of a minimum number of training rounds required for HP search space reduction; an indication of a size of a data partition of a dataset to be used for HP search space reduction; or an indication of minimum data coverage or distribution percentage for a particular feature in a data partition of a dataset to be used for HP search space reduction.

19. The method recited in claim 17, further comprising receiving, from the FL service, an identifier of a RESTful resource storing at least (i) the one or more requirements sent by the FL user device in the request for HP search space reduction, and (ii) information associated with one or more partitioned datasets of the plurality of FL clients.

20. The method of claim 17, wherein the information indicative of the results of the HP search space reduction training performed by the plurality of FL clients that is received by the FL user device comprises one or more of: the identifier associated with the FL training of the machine learning model; a list of best performing HP configurations resulting from the HP search space reduction training; statistics associated with each of one or more HP configuration; or one or more recommendations of HP configurations to pursue in connection with FL HP optimization.