EP4690013A1

EP4690013A1 - Collaborative model composition and reusability via split learning

Info

Publication number: EP4690013A1
Application number: EP24785435.9A
Authority: EP
Inventors: Mirko D'ANGELO; Selim ICKIN; Alexandros PALAIOS; Hannes LARSSON
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2023-04-07
Filing date: 2024-03-26
Publication date: 2026-02-11
Also published as: WO2024210792A1

Abstract

A computer‐implemented method of training a machine learning (ML) model using split learning includes providing a first ML model that receives a set of input values (X) and 5 generates a set of output activations (X') in response to the input values. The output activations are provided as inputs to a plurality of second ML models. Each of the second ML models generates a different output (y) based on the output activations generated by the first ML model. The first ML model and the plurality of second ML models are jointly trained using the set of output activations. The method generalizes model weights of the first ML model 10 such that the first ML model outputs representable features in the form of activations for different tasks implemented at the plurality of second ML models.

Description

COLLABORATIVE MODEL COMPOSITION AND REUSABILITY VIA SPLIT LEARNING

TECHNICAL FIELD

[0001] The present inventive concepts relate to distributed machine learning and in particular to split learning systems for training a machine learning model.

BACKGROUND

[0002] The management of a complex system, such as a wireless communication network, can be aided by employing machine learning (ML) models to analyze the system and select appropriate control actions to take on the system. For example, ML models can be used to track and control system operations and/or performance metrics. In a wireless communication network, these operations/metrics may include such things as handover, throughput, latency, and others.

[0003] To train an ML model, it is desirable to use a large amount of training data. In a wireless communication network, such training data may include network analytics data that is drawn from a number of sources, such as base stations, switches, relays, servers and other network equipment, as well as from user equipment that is operating in the network.

[0004] The use of analytics data from various parts of a network may raise privacy concerns, as it may not be desirable to share certain analytics data across a network. Moreover, due to the large amount of data that is needed for training, it may be desirable to use a collaborative training approach, such as federated learning or split learning, in which different parts of an ML model are trained by different entities. Such learning techniques may also raise data privacy concerns, however.

[0005] In a wireless communication network, split learning may be performed in a network data analytics function (NWDAF). Figure 1A illustrates a hierarchical NWDAF deployment in a wireless communication network. In particular, Figure 1A shows how the NWDAF architecture might take the form of a hierarchical NWDAF deployment, allowing distributed ML techniques such as federated learning (FL) and Split Learning (SL) to take place. In particular, a hierarchical NWDAF deployment may include a centralized NWDAF and one or more distributed NWDAFs in different areas along with local core (5GC) network functions (NF).

[0006] Federated learning is a machine learning approach that enables multiple devices or systems to collaboratively train a shared model without centralizing the data. Instead of collecting all the data to a central server, the training data is kept on the local devices, and the updates to the model are sent back to the central server. This helps to preserve the privacy of the training data.

[0007] An example of a federated learning system is shown in Figure IB. As shown therein, a plurality of training nodes (Training node 1, training node, 2, etc.) train local copies of a shared model Ml using local datasets (Dataset 1, Dataset 2, etc.). Weights generated by the training node are transmitted to a central server that combines the weights and sends the combined weights back to the training nodes.

[0008] Each training node trains the model on its own data and sends only the updated model weights back to the central server, where they are combined with other updates to improve the generic ML model. This process may be repeated multiple times until the model achieves a desired accuracy.

[0009] Federated learning offers several benefits, including increased privacy, reduced communication overhead, and the ability to scale to large distributed systems. It is particularly useful in scenarios where data privacy is a concern, such as healthcare, finance, and other sensitive domains.

[0010] Split learning is a machine learning approach that enables the training of deep neural networks in a distributed and privacy-preserving manner. In SL, the model may be divided into two parts: the client-side and the server-side along a so-called cut-layer. The cutlayer is the last layer of the client-side model and the first layer of the server-side model. It is the layer that interfaces the client-side and server-side neural network models. The client-side is responsible for processing the input data and generating a set of intermediate representations. These representations are then securely transmitted to the server-side, where they are combined with the rest of the model to produce the final output. [0011] A SL arrangement involving a single client is illustrated in Figure 1C, and a split learning arrangement involving multiple clients is illustrated in Figure ID.

[0012] Referring to Figure 1C, in an example of SL involving a single client, a client ML model performs forward-pass on its input dataset (X) and produces an encoded version of its input X'. This may be expressed as M: X --> X', where X', referred to as activations, represent a so-called "smashed" form of X (transformed over neural networks). The dimensionality of the activations X' can be decreased relative to X with the number of neurons interfacing the clients to the server node.

[0013] The server receives the encoded activations X' and performs a forward pass on its own local neural network to obtain outputs y', M: X' --> y'. In the case of multiple clients, then X' from the individual clients may be concatenated . During training, the server has label information, meaning that it can compute the loss (y' - y), where y is the real label and y' is the estimated output of the server model. The server then performs a backward pass (backpropagation) to obtain the gradients at its cut-layer interface to the client, i.e., grad_client_l = backward(error). The client receives the gradients from the server and performs a backward-pass on the gradients, i.e., performs back propagation of gradients toward the client ML model's first layer. The neural network weights then are updated. This completes a round of training, but there can be as many rounds as needed until the model converges, i.e. when the error between y' and y reduces to a predetermined level or changes less than a predetermined amount.

[0014] The split learning approach is useful when the data is sensitive, and it is not desirable to transfer it to a central server for processing. Instead, the processing of input data is performed locally on the client-side, and only the intermediate representations are shared with the server-side, preserving the privacy of the data.

[0015] Split learning may be particularly useful in scenarios where there is a large number of clients with limited computing power, such as mobile devices, and the central server has ample computing resources. By distributing the computation, split learning allows for efficient use of resources and enables the training of large models on a large number of clients. [0016] In the context of a communication system, such as a 5G communication system, a split architecture may be provided with part of the model being placed in a NWDAF and part of the model being hosted in an application function (AF). In this case, similar to Figure 1A, application functions (AF) might be placed at the leaves of the split architecture, with a centralized NWDAF residing in the network providing part of the ML model. Moreover, a part of the model can be placed in one NWDAF, while the other part is placed in another NWDAF.

[0017] In SL involving multiple clients as shown in Figure ID, the head model is split into multiple sections Head Model 1, Head Model 2, etc., and each head model is trained on a different client. Each head model may receive different inputs (Xi, X2) and generate different activations X'i, X'2 which are input to different portions of the tail ML model at the cut-layer. In the example, the tail nodes share the same target variable, y, hence two tail nodes are combined. In addition, in the case the tail nodes may not share the same target variable, y, then two SL model weights can be aggregated and collaboratively trained via federated learning.

[0018] It has been shown that SL may yield significantly more efficient model training on large neural network models than FL with respect to computation and energy. There are a number of possible explanations for this. First, in SL, model weights are not shared. Instead, the activations (i.e. the encoded outputs of neural network models at the cut-layer) are shared. Second, in sequential SL, the training is sequential, meaning that the collaborating clients that are not training can be shut down. This may increase the idle period of a client and thereby save energy when it is not being used for other purposes. There is also parallel split learning that allows simultaneous joint training by activation concatenation and gradient aggregation.

[0019] SL may also allow full or partial computation and model training offload in between clients, and may allow for distributed cascaded model training.

SUMMARY

[0020] As discussed above, mechanisms that reduce threats to end-user privacy are desirable to facilitate collaborative machine learning in many fields, such as healthcare, finance and communications. In some applications, different ML models are needed for different tasks. However, even models that are designed for different purposes may nevertheless utilize the same underlying system analytics as inputs. To improve efficiency, it would be desirable to be able to re-use analytics data in different ML models, even if the models are trained for different purposes. In particular, the ability to re-use analytics may be important for efficiently generating ML models in highly complex environments, such as in a 5G core network (5GC) and subsequent evolutions (e.g., 6GC).

[0021] Accordingly, some embodiments provide a computer-implemented method of training a machine learning (ML) model using split learning that includes providing a first ML model that receives a set of input values (X) and generates a set of output activations (X') in response to the input values. The output activations are provided as inputs to a plurality of second ML models, wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model.

[0022] The method then jointly trains the first ML model and the plurality of second ML models using the set of output activations. Jointly training the first ML model and the plurality of second ML models includes simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training. The method generalizes model weights of the first ML model such that the first ML model outputs representable features in the form of activations for different tasks implemented at the plurality of second ML models.

[0023] The method may further include generating, at the plurality of second ML models, respective outputs related to the different tasks implemented at the plurality of second ML models.

[0024] The tasks implemented at the plurality of second ML models may include generating predictions of network performance measurements and/or network events in a communication network.

[0025] The network performance measurements and/or network events may include throughput, handovers, call drops, physical resource block utilization, channel quality degradation, and/or latency in the communication network.

[0026] Simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training may include aggregating gradients from the plurality of second ML models to provide aggregated gradients and performing a backward pass on the first ML model using the aggregated gradients.

[0027] The first ML model may include a first neural network comprising a first plurality of layers, and wherein the second ML models may include respective second neural networks.

[0028] The intermediate activations may include outputs generated at an output layer of the first neural network.

[0029] Each of the second ML models may be configured to perform a different estimation function. The different functions may include estimation functions or prediction functions.

[0030] The input values may include performance monitor counters for a communication network, and the estimation functions may perform estimation of at least one of, but not limited to, handover, throughput, call drop, physical resource block utilization, channel quality degradation, and/or latency in the communication network.

[0031] Jointly training the first ML model and the set of second ML models may include performing initial training on the first ML model.

[0032] The initial training may be performed jointly with at least one of the second ML models. For example, the initial training may be performed together with a set of the second ML models in sequential phases.

[0033] The first ML model may be trained together with each of the set of second ML models as an overall ML model in sequential phases until convergence of the overall ML model in each phase.

[0034] Jointly training the first ML model and the set of second ML models may include training the first ML model in an initial training phase followed by training the first ML model and the set of second ML models in a model generalization phase.

[0035] Training the first ML model and the plurality of second ML models in the initial training phase may include training the first ML model together with each of the second ML models in sequential phases until convergence.

[0036] Training the first ML model and the plurality of second ML models in the model generalization phase may include, for each of the plurality of second ML models, sequentially training the first ML model and the second ML model together for one round, performing a forward pass on the first ML model, recording the activations generated by the first ML model, and obtaining a loss of the second ML model, aggregating gradients of the second ML models, performing backpropagation on the first ML model using the aggregated gradients, and updating the first ML model and the second ML models based on the results of the backpropagation.

[0037] The operations of sequentially training the first ML model and each of the plurality of second ML models for one round and updating the first ML model and the second ML models after each round may be performed until convergence of the first ML model and the second ML models.

[0038] The method may further include providing the activations generated by the first ML model after training the first ML in the initial training phase to a use case classifier model, and training the use case classifier model to predict activations belonging to each of the second ML models.

[0039] The method may further include, in the model generalization phase, at each round of joint training of the first ML and one of the second ML models, training the use case classifier for one round with activations generated at the round of joint training, computing a cross entropy loss at an output of the use case classifier model, obtaining gradients of the use case classifier model, reversing signs of the gradients of the use case classifier model to obtain sign-reversed gradients, providing the sign-reversed gradients of the use case classifier model to the first ML model, and aggregating the sign-reversed gradients of the use case classifier model with the gradients of the second ML models before performing backpropagation on the first ML model.

[0040] The method may further include calculating a loss function at each backpropagation operation in the generalized training round, and regularizing losses of the second ML models based on the loss function. The loss function may compute a difference between first ML model weights between two consecutive split learning rounds.

[0041] The loss function may include a first component that computes a difference between first ML model weights between two consecutive split learning rounds and a second component that computes a loss of a second ML model, wherein the first component and the second component are combined to calculated losses when performing backpropagation of the models. The first component and the second component may include mean square error loss functions.

[0042] The first component may be given by the equation:

MS E GenericWeight Loss, n ~ Q ~ Q ) wherein Qⁿ is a weight matrix of the first ML model at training round n, and the second component may be given by the equation: wherein i is a sample index and y¹ is an actual output of the second ML model for which the loss function is evaluated, and y¹' is an estimated output of the second ML model.

[0043] A computer implemented method performed by a client network function in a communication network for training a split ML model, is provided, where the split ML model comprises a second ML model of the split ML model. The method includes transmitting a discovery request to a discovery function in the communication network, wherein the discovery request requests an identity of a data analytics function in the communication network that manages first models, receiving from the data analytics function, information indicating whether a first model managed by the data analytics function is suitable for training the second ML model, and initiating split training with the data analytics function of a ML model including the second ML model and the first ML model based on the information received from the data analytics function.

[0044] The discovery request may include analytics metadata relating to the second ML model.

[0045] The information received from the data analytics function may include a training relation graph that indicates suitability of the first ML model for training with the second ML model. [0046] The information received from the data analytics function may include an estimated gain associated with training the second ML model with the first ML model.

[0047] The communication network may include a core network of a wireless communication system, and the data analytics function may include a network data analytics function, NWDAF, of the core network. The client network function may include a NWDAF. The data analytics function may host a model training logical function that performs joint training of the first ML model and the second ML models.

[0048] The split training of the ML model may include joint training of the first ML model with a plurality of split training clients, each of which trains an associated second ML model.

[0049] The first ML model may receive a set of input values (X) and generate a set of output activations (X') in response to the input values, wherein the output activations are provided as inputs to the second ML models associated with the split training clients, wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model. Joint training may include jointly training the first ML model with the split training clients using the set of output activations.

[0050] A computing device according to some embodiments includes a processor circuit and a memory coupled to the processor circuit. The memory stores computer program instructions that, when executed by the processor circuit, cause the computing device to perform operations including providing a first ML model that receives a set of input values (X) and generates a set of output activations (X') in response to the input values. The output activations are provided as inputs to a plurality of second ML models, wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model.

[0051] The computing device then jointly trains the first ML model and the plurality of second ML models using the set of output activations. Jointly training the first ML model and the plurality of second ML models includes simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training. The computing device generalizes model weights of the first ML model such that the first ML model outputs representable features in the form of activations for different tasks implemented at the plurality of second ML models.

[0052] A computer program product according to some embodiments includes a computer-readable storage medium that stores computer program instructions that, when executed by a processor circuit of a computing device, cause the computing device to perform operations including providing a first ML model that receives a set of input values (X) and generates a set of output activations (X') in response to the input values. The output activations are provided as inputs to a plurality of second ML models, wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model.

[0053] The computing device then jointly trains the first ML model and the plurality of second ML models using the set of output activations. Jointly training the first ML model and the plurality of second ML models includes simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training. The computing device generalizes model weights of the first ML model such that the first ML model outputs representable features in the form of activations for different tasks implemented at the plurality of second ML models.

[0054] Methods of training an ML model as described herein may provide certain advantages. For example, methods according to some embodiments may increase the energy efficiency of training while maintaining or improving data privacy. Moreover, the use of a generic head model may reduce the amount of data collection required to perform training of multiple use case models. Additionally, split learning techniques according to some embodiments may enable the reuse of pre-trained generic models to build more complex models in cascaded form, potentially resulting in a more scalable and/or extensible network management system.

BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1A illustrates a hierarchical network data analytics function (NWDAF) deployment in a public land mobile radio network (PLMN). [0056] Figure IB illustrates a federated learning arrangement for machine learning.

[0057] Figure 1C and ID illustrates a split learning arrangement for machine learning.

[0058] Figures 2A and 2B illustrate split learning arrangements for machine learning according to some embodiments.

[0059] Figure 3 illustrates a network function service registration procedure.

[0060] Figure 4 illustrates network function service discovery in the same PLMN.

[0061] Figure 5 illustrates collaborative training.

[0062] Figures 6 and 7 are graphs that illustrate an example of a result of sequential SL.

[0063] Figure 8 illustrates systems/methods for performing hybrid SL according to some embodiments.

[0064] Figure 9 illustrates systems/methods for performing hybrid SL with a UC classifier according to some embodiments.

[0065] Figures 10A, 10B and 10C summarize a comparison of standalone split learning training versus the Methods 1, 2, 3 and 4 for the three use cases.

[0066] Figures 11A and 11B illustrate a procedure for the training of a generic ML model in hybrid split learning with UC classifier using a combination of sequential and parallel training with gradient reversal (Method 3).

[0067] Figures 12 and 13 illustrate operations for SL according to some embodiments.

[0068] Figure 14A is a block diagram illustrating elements of a data analytics function according to some embodiments.

[0069] Figure 14B illustrates various functional modules stored in the memory of the data analytics function according to some embodiments.

DETAILED DESCRIPTION OF EMBODIMENTS

[0070] Training different parts of a neural network for different tasks is typically done by re-training the last few layers of a neural network, which are seen as task-specific, while the first layers are more related to extracting features of the input data.

[0071] There is a technique in decentralized learning known as "incremental task learning," in which one neural network can be trained on multiple tasks sequentially. Although

11

SUBSTITUTE SHEET (Rule 26) this may help to reduce training time for multiple tasks, the results might still be suboptimal for the individual use cases. Multi-task learning is another related area in which the same neural network can learn multiple tasks. This is done by having extracting multiple outputs from a neural network. Typically, the first layers of the neural network are shared for all tasks, but the last layers are specific for each task. This is typically done in parallel, and typically requires that the target variables for different tasks exist in the same node or client.

[0072] Hard negative sampling and contrastive regularization may be used to enable generic model training along with the personalized client models in a federated learning setting. However, this necessitates some global data to be shared with the client nodes.

[0073] Some embodiments provide systems/method for performing split learning that allows generic ML models to be re-used across multiple use cases. In particular, some embodiments train a generic neural network (NN) model that is used as a base component for multiple application-specific NN models. For example, according to some embodiments, a NN model may be split along a cut-layer into a generic portion as the head model and one or more specialized portions as tail ML models. A server can train the generic portion of the ML model, which encodes a common output layer. One or more client ML models can be trained to transform the encoded common output layer to perform specialized tasks.

[0074] This approach may allow a generic ML model to be re-used sequentially by different tail ML models, while also keeping target variables for different tasks in different client nodes.

[0075] In this scope, a client node can be an NWDAF or another AF. However, in general, the concept can apply to generic radio access network (RAN) core network functions (NF). In a SL environment involving multiple clients, none of the clients training the model can see the data of another client.

[0076] A distributed ML model training topology according to some embodiments is illustrated in Figure 2A. On the left-hand side of Figure 2A, a centralized function 20, such as a NWDAF, hosts a generic ML model 22 (e.g., a neural network), that takes input features X and outputs encoded versions X' of the input features. Multiple clients 24-1, 24-2, 24-N on the right-hand side have different use cases (UCs) or tasks to perform. Thus, each client trains a different tail ML model for performing its specific task (UC 1, UC 2, etc.). Each client may transform the compressed input provided by the generic ML model into different outputs (yi, y?, YN) depending on their task.

[0077] The generic ML model 22 is trained along with each of the UC-specific tail ML models (UC 1 tail ML model, UC 2 tail ML model, etc.). At the end of the training, the generic ML model 22 has weights that may be reusable for multiple use cases. That is, the generic ML model 22, which corresponds to the first several layers of the overall ML model, is reusable with multiple tail ML models, and the remaining layers of the tail ML models, which are located in the specialized use case nodes, are personalized with respect to a UC-specific loss function (yn, y'n).

[0078] Figure 2B illustrates an embodiment in which multiple different generic ML models are trained in different regions (e.g., Region 1, Region 2) via a federated learning arrangement. A particular UC node may use whichever generic ML model is suitable for its needs.

[0079] There exist many similar use cases in communication systems that use the same input variables, such as the same performance measurement counters collected by network nodes, but that have different target goals/tasks to accomplish. It is often the case that use cases implemented for key performance indicator (KPI) prediction tasks, such as latency prediction, throughput prediction, call drop prediction, CQI degradation prediction, etc., take all available input performance monitor (PM) counters as inputs. In all cases, the ML models are often separately trained. For example, for latency prediction and throughput prediction use cases, many PM counters are provided as inputs to individual models that have different target variables. Thus, large portions of the different neural network models may be very similar, and it would be a waste energy and computation resources to perform independent training of multiple neural network models for very similar use cases.

[0080] Some embodiments have certain advantages over conventional state-of-the-art SL approaches with respect to storage, as a single generic model according to some embodiments provides inputs to multiple tail models. Moreover, there may be significantly less need for maintaining multiple generic models to provide scalability. Although, the computation complexity may increase, in some cases, R (the number of required rounds until ML model convergence) may be reduced significantly using methods described herein, which may reduce the network footprint and computation overhead. Some embodiments may also reduce the network footprint needed for SL training, while also improving accuracy and/or reducing convergence times.

[0081] In the context of a 5G communication system, some embodiments provide systems/methods that enable model training of a generic ML model in a NWDAF, via split learning in a decentralized manner, for different use cases between clients. The clients can be NFs, application functions (Afs), and/or other NWDAFs. The systems/methods may recommend existing pre-trained generic ML models to clients that might potentially benefit from using the generic ML model depending on their respective requirements. The requirements of specific clients may include ML model serving time, energy consumption, model accuracy, etc.

[0082] In a SL model according to some embodiments, the generic ML model can be federated (e.g., via horizontal federated learning) within different regions (e.g., amongst different NWDAF).

[0083] Accordingly, systems/methods according to some embodiments may provide ML models that can be trained with increased privacy and/or energy efficiency. The use of reusable generic components within different ML models may alleviate costly data collection procedures. Moreover, split learning enables the reuse of pre-trained ML models to build more complex ML models in cascaded form, which may provide a more scalable and extensible network system.

[0084] Embodiments will now be described with reference to a 5G communication network. However, it will be appreciated that embodiments described herein may be applicable to many different types of systems in which machine learning techniques maybe employed.

[0085] In development of a ML model that is generic to multiple use cases with SL, a distributed ML model may be split into multiple parts (e.g., to a server and a plurality of clients). A head model corresponding to the generic portion of the ML model resides in a central server, while the other parts of the ML model, namely tail ML models that are personalized to specific use cases, reside in a plurality of client computing devices. The generic ML model is trained jointly, either in an alternating sequential manner and/or a hybrid (sequential and parallel) manner via split learning, with many different use case ML models (i.e., clients) that each have different tasks, or use cases. The generic ML model generalizes common weights and can be reused in multiple use cases.

[0086] Each time a new client requests a generic ML model, the server may determine whether an existing generic ML model is suitable for use with the new use case. If not, then the server may request ML model retraining, such as via a model training logical function (MTLF) in an NWDAF setting, to update the generic ML model so that it may be used with the newly requested use case ML model. The trained generic ML model is then sent to the requesting client, and meta information that contains coverage information of the generic ML model may be updated.

[0087] Systems/methods according to some embodiments will now be described in the context of phases of registration, discovery, collaborative model training, and model inference.

[0088] Registration and analytic metadata exchange

[0089] Registration may be performed in the following way for NWDAFs containing an MTLF providing support for training a generic ML model, and hence act as the server of the split NN model architecture.

[0090] The NWDAF may register its network function (NF) profile, the address of the NWDAF, and support of server split model training capability via a network repository function (NRF).

[0091] Figure 3 shows a procedure in which a NF service consumer is an NWDAF having an MTLF supporting server split model training capability. The procedure allows clients (NWDAFs or AFs) to discover generic ML models available in the network. Note that many such generic ML models might exist, and clients may join to the one that generalizes to its use case the best. In a simple network configuration, only one NWDAF registers itself as a server supporting split model training, and all the clients of the split architecture will collectively train jointly such generic ML model. [0092] NWDAFs that participate in the split learning procedure as clients and not servers may register following a NF service consumer to NRF registration procedure. For example, referring to Figure 3, the NWDAF may send an NRF registration request to an NRF (step 1). The NRF stores the NWDAF profile in step 2 and sends an NRF registration response to the NWDAF at step 3.

[0093] Registered NWDAFs, both clients and servers, may support the request and exchange of "analytics metadata information" that allows exchanging information related to the produced model output analytics. The content includes, for example, one or more of (i) the number of data samples for the generation of the analytics, which includes both the number of rows of a dataset and the number of columns, (ii) data source(s) of the data used for the generation of the analytics, (iii) the distribution with respect to historical traces of the accuracy of the generic ML model when trained on multiple use case ML models, (iv) the distribution with respect to historical traces of the energy consumption of the generic ML model when trained on multiple use case ML models, and/or (v) the distribution with respect to historical traces of the time-to-serve metric of the generic ML model. Time to serve is measured as the time difference between the time of the request initiated from the client to the server and the time it was served to the client from the server.

[0094] The exchange of analytic metadata information may allow a server NWDAF to make cost-aware decisions when collectively training a generic ML model for multiple clients.

[0095] Application functions (AFs) that are trusted (i.e., native, such as V2X applications and XR applications) may follow the same registration and discovery procedure of NWDAFs. That is, they may use the NRF for communications with the core network. For untrusted applications, (such as Netflix, YouTube, banking apps, etc.) communication is realized by using the Network Exposure Function (NEF). In the following description, only the flow for client NWDAFs will be illustrated. However, it will be understood that the same flows will apply to other AFs, with the exception that NEF will mediate the communication with the core network in such cases.

[0096] Note that while NWDAF and AF can be both clients for the split learning procedure, only NWDAFs can be heads of generic ML models. [0097] Discovery

[0098] A specific client ML model residing in the AF or NWDAF may discover a NWDAF that supports split model training capability using a discovery procedure. Referring to Figure 4, an NRF discovery request (Nnrf_NFDiscovery_Request) is transmitted to the NRF for discovery in step 1.

[0099] The discovery request includes an indication of a NWDAF, containing MTLF, that supports split model training capabilities as a server. After authorizing the request in step 2, the NRF may return, in an NRF discovery response (step 3), multiple NWDAFs candidates matching the requested capabilities, because many NWDAFs might exist that support server split model training as servers.

[0100] For an AF, the network exposure function (NEF) is responsible for controlling the analytic exposure mapping of network ML models, so the AFs do not contact the NRF directly, but instead may use analytic exposure capabilities provided by an NEF for discovering the server split model training capable NWDAFs.

[0101] Split training procedure

[0102] A client needs to make sure that the accuracy of its ML model would benefit from the generic ML model that has been trained with other use case ML models. In some cases, a use case ML model can be trained standalone with the generic ML model (i.e., without joint training with other use case ML models). In other cases, a use case ML model can be trained jointly first with the generic ML model, and can then reuse a generic ML model that was previously trained jointly with other use case ML models. In a joint training with multiple use case ML models approach, the systems/methods ensure that the base generic ML model is trained in a way that it is use case agnostic and will benefit multiple use case ML models simultaneously. This may improve the efficiency of training.

[0103] To formulate the training problem, on the one hand, it is desired to reduce/minimize the number of generic ML models, and on the other hand, to increase/maximize the individual accuracy of the use case ML models. [0104] Figure 5 shows a procedure for joining an available NWDAF that is a server of generic ML models. Referring to Figure 5, in step 1, an NWDAF client sends a discovery request to the NRF to identify an NWDAF that supports split training as a server.

[0105] In step 2, a list of NWDAF that manage generic MTLF models is returned by the NRF (in this example NWDAF 1 and NWDAF 2). The NWDAF client wants to join the server model that provides the best generalizability, and that also provides complementary updates to improve the joint model further, for its use case.

[0106] To this end, in steps 3 and 4, the NWDAF client sends evaluate split training requests to the plurality (in this case two) identified servers. The request includes its analytic metadata information. Examples of analytic metadata information include the number of data samples used for generation of output analytics, the data sources(s) of the data used for the generation of the analytics, data formatting and processing applied on the data, features used, data distribution and statistics, etc.

[0107] At this point, the NWDAF servers have statistics from the historical requests from other consumers and training process with other consumers. In response, the NWDAF servers estimate whether or not the existing generic ML model would benefit the client, and appends an estimated gain in the footprint at the consumer side. Based on the estimated gain, the client can request training at the NWDAF or at the client. Accordingly, in steps 5 and 6 the server NWDAFs may generate training relation (i.e., similarity) graphs based on the analytics metadata information received by the client. This can be performed, for example, based on a causal inference (understanding the impact of one use case ML model on the generic ML model).

[0108] In another example, the training relation graphs may be generated based on the similarity of the analytic metadata information shared by the client and the existing metadata library of other clients and/or using recommendation engines, such as collaborative filtering or generative models. The use of training similarity graphs can be realized by using well known methods.

[0109] In steps 7 and 8, the NWDAF servers send feedback responses to the NWDAF client. The feedback can include information such as the training relation graphs, which indicate the extent to which the client would benefit from the generic ML model and contribute to overall model training.

[0110] In step 9, the NWDAF client joins the generic ML model training that brings the most benefit to its use case (but also improves the existing generic ML model). In this example, the NWDAF client joins NWDAF Server 1 (as it was found to be the most suitable).

[0111] In step 10, NWDAF Server 1 registers the NWDAF client to the joint training.

[0112] If no suitable graph is reported by the available NWDAF servers, the NWDAF client may train both the use case ML model and the generic ML model in an SL setting in isolation, and then may register the generic ML model together with the meta data. Even if the model is trained in an isolation at the same client, the split aspect is important for registering the generic portion of the ML model separately and reusing/recommending it for other use case ML models.

[0113] Once discovery is performed, a client ML model can join the split learning group that is managed by a NWDAF (containing MTLF) acting as a server model trainer. The flow shown in Figure 5 is shown for NWDAFs clients. A similar flow may apply to AF clients, except that information may be exchanged between the AF client and the NWDAF server(s) via NEF.

[0114] The training of the generic and use case ML models via split learning can be performed using a number of different methods according to various embodiments. Several methods will now be described, starting from the simplest and gradually increasing in complexity.

[0115] Different approaches for SL training in systems/methods according to some embodiments will now be described. In each of these cases, it is assumed that there is one generic ML model and two use cases, UC 1 and UC 2, for which models are trained by two respective clients, Client 1 and Client 2.

[0116] Baseline: Standalone SL training of multiple use cases

[0117] In a baseline scenario, the UC models are trained in standalone manner. That is, each UC model is trained with the generic ML model separately, resulting in a different "generic" model for each UC. [0118] From the perspective of model reusability, having different generic ML models for each use case is not efficient, since this would increase the storage requirements in the model management system.

[0119] Method 1: Generic ML model with Sequential Split Learning

[0120] In sequential split learning, the server (head) model is trained sequentially with each individual use case (tail) model Referring to the topology shown in Figure 2A, the generic ML model 22 is trained with the use case ML models in a sequential manner as described below. First the generic ML model is trained with Client 1, then Client 2, and so on. Once all clients have trained with the generic ML model, one round of SL training is completed. This may be repeated over multiple rounds.

[0121] Figure 6 illustrates a loss curve for sequential training over 8 training rounds where the training process switches from one client, i.e., one use case, to another one alternatively. That is, the generic ML model is trained with UC 1 ML model for a number of rounds (602), and then trained with UC 2 ML model for a number of rounds (604). The example illustrated in Figure 6 represents an ideal case in which the training datasets have an independent and identically distributed (iid) distributions. In that case, there is a clear convergence in both training phases: training phase on client 1 (UC1) and then the training phase on client 2 (UC2).

[0122] A beneficial effect of sequential SL is that it enables the offload of ML models to multiple clients, which helps to reduce computation cost at one single client.

[0123] However, in reality, the datasets may be non-iid, and the tasks can be significantly different from each other. In the case when datasets are non-iid, such as when the use cases are very different, sequential split learning might suffer from a phenomenon called catastrophic interference (or catastrophic forgetting). This is illustrated at arrows 704 in Figure 7, which is a graph of training score at each SL training round for a single client (UC 1). At the end of the first rounds (at about round 150), the ML models converge to a high accuracy value (r2-score), and then at the next round, the generic ML model is trained with the other use cases, hence the model accuracy drops catastrophically. [0124] Accordingly, some embodiments described below provide different training approaches using a hybrid split learning approach which is a combination of sequential and parallel SL. In particular, a Hybrid SL training approach according to some embodiments includes sequential SL followed by parallel (or quasi-pa ra I lei) SL.

[0125] Method 2: Hybrid Split Learning

[0126] Systems/methods for training a neural network using hybrid SL are illustrated in Figure 8. In a hybrid SL approach according to some embodiments, an initial training phase is provided in which (in an example with UC 1 and UC 2), the first use case (UC 1) tail ML model is trained with the generic ML model for some rounds until the overall model converges, and then the second use case (UC 2) tail ML model is trained with the generic ML model for some rounds until the overall model converges.

[0127] After the initial training phase, a model generalization phase is performed. At this point, all models (e.g., the generic ML model, UC 1 tail ML model, UC 2 tail ML model) have been pre-trained. An objective of the model generalization phase is to have joint training while keeping the generic ML model as generalizable as possible.

[0128] Referring to Figure 8, in the model generalization phase, UC 1 tail ML model is trained with the generic ML model for one additional round, after which a forward-pass (step 1) is performed on the generic ML model, and the activations X' generated by the generic ML model are recorded.

[0129] The method then computes and obtains the loss (y - yi) of the UC 1 tail ML model (step 2) and calculates gradients for the UC 1 tail ML model, which are sent to the generic ML model 22 at step 2a.

[0130] The UC 2 tail ML model is then trained with the generic ML model for one additional round, and the method performs a forward-pass on the generic ML model and records the activations X' generated by the generic ML model (step 3).

[0131] The method then computes and obtains the loss (y₂' - y₂) of the UC 2 tail ML model (step 4) and calculates gradients for the UC 2 tail ML model, which are sent to the generic ML model 22 at step 4a. The gradients received from the UC 1 tail ML model and the UC 2 tail ML model are then aggregated or combined (step 5). [0132] The method then performs backwards propagation of the aggregated gradients at the generic ML model 22 (step 6), and updates all models.

[0133] The steps of the model generalization phase are then repeated until ML model convergence.

[0134] Method 3: Hybrid Split Learning with UC Classifier

[0135] A hybrid SL with UC classifier approach according to some embodiments is illustrated in Figure 9, in which two tail ML models, UC 1 and UC 2, are trained with a generic ML model 22. In the hybrid SL with UC classifier approach, an initial training phase is provided, followed by a model generalization phase. For ease of notation, the activations generated by the generic ML model following training with a UC are represented in the following discussion as "a."

[0136] Initial Training Phase

[0137] Referring to Figure 9, in the initial training phase, the UC 1 tail ML model is trained with the generic ML model 22 for some number of rounds until the overall model converges.

[0138] The activations ai of the generic ML model 22 at the last round of training with the UC 1 tail ML model are recorded and labelled as belonging to UC 1.

[0139] The UC 2 tail ML model is then trained with the generic ML model 22 for some rounds until the overall ML model converges.

[0140] The activations a2 of the generic ML model 22 at the last round of training with the UC 2 tail ML model are then recorded and labelled as belonging to UC 2.

[0141] The activations and their labels are sent to a UC classifier 36. The UC classifier 36 trains its own NN model for some rounds until the model converges.

[0142] Model Generalization Phase

[0143] Following the initial training phase, all of the ML models (the generic ML model 22, the UC 1 tail ML model, the UC 2 tail ML model, and the UC classifier 36) have been pretrained. In the model generalization phase, the UC 1 tail ML model is trained with the generic ML model 22 for one additional round, and a forward-pass is performed the on the generic ML model 22. The resulting activations ai are recorded. [0144] The method then computes and obtains the loss of the UC 1 tail ML model.

[0145] The UC 2 tail ML model is then trained with the generic ML model 22 for one additional round, and a forward-pass is performed on the generic ML model 22. The resulting activations a₂ are recorded.

[0146] The method then computes and obtains the loss of the UC 2 tail ML model.

[0147] The method then sends the recorded activations generated from training the UC 1 tail ML model and the UC 2 tail ML model to the UC classifier 36. In the forward propagation operation, the gradient reversal layer 34 acts just as a proxy and outputs the activations directly to the UC Classifier 36.

[0148] The UC classifier 36 is trained for one round with the received activations. The corresponding labels indicate which activations belong to which use case (e.g., labekl for UC 1 and label:2 for UC2).

[0149] A cross entropy loss at the output of UC classifier 36 is then calculated. Using the cross-entropy loss, the UC classifier 36 learns to distinguish between activations that are input to different use cases. This operation is reversed by the gradient reversal layer 34, and the "reversed" (or "sign altered") gradients are sent to the generic ML model 22 to be used in backward propagation so that the generic ML model 22 becomes agnostic to the use cases. This may help the generic ML model 22 to generate activations that are common to all use cases.

[0150] The method then obtains the gradients g of the UC classifier 36, and reverses the sign of the gradients (via a gradient reversal layer 34) before sending the reversed gradients greversed to the generic ML model 22.

[0151] The method then aggregates the gradients gi, g₂ received from the UC 1 tail ML model and the UC 2 tail ML model, and the reversed gradients greversed from the UC classifier 36.

[0152] Finally, the method performs backwards propagation at the generic ML model 22, and updates the weights of all ML models.

[0153] The steps of the model generalization phase are repeated until ML model convergence.

[0154] Method 4: Hybrid Split Learning with Regularized Loss [0155] In Method 4, the training methodology is very similar to Methods 2 and 3. The gradients in Method 4 are computed in consideration of the following two loss functions. where i is the sample index, n is the SL round index, and Q is the weight matrix of the generic ML model at round n. Equation [1] computes the difference of the generic ML model weights between two consecutive split learning rounds. Equation [2] is the estimation/prediction loss of a use case.

[0156] The loss functions shown above are aggregated when performing one full backward propagation. This may prevent the generic ML model from changing too much over each training round while still helping the client ML model to jointly train the generic ML model. The objective is to reduce/minimize both the estimation loss and the amount of change of the generic ML model over each training round, simultaneously. During the backward-propagation of the generic and UC models, the two loss functions are aggregated (e.g., as a weighted sum where weights are equal, i.e., 0.5) so that the gradients are changed with respect to the aggregated loss.

[0157] As with the above methods, Method 4 includes an initial training phase followed by a model generalization phase, described below in an example including two use cases, UC 1 and UC 2.

[0158] Initial Training Phase

[0159] In the initial training phase, the UC 1 tail ML model is trained with the generic ML model for some rounds until the overall ML model converges.

[0160] The UC 2 tail ML model is then trained with the generic ML model for some rounds until the overall ML model converges.

[0161] Model Generalization Phase

[0162] Upon completion of the initial training phase, all ML models (e.g., the generic ML model 22, UC 1 tail ML model, UC 2 tail ML model are pre-trained. To make the generic ML model as generalizable as possible, in the model generalization phase, a forward-pass is performed on the generic ML model 22, and the activations of the generic ML model 22 are recorded. A forward pass is then performed on the UC tail ML models.

[0163] The method then computes and obtain loss at the UC 1 tail ML model, and computes and obtains the loss at the UC 2 tail ML model.

[0164] The method then aggregates the gradients received from the UC 1 tail ML model, the UC 2 tail ML model, and the weight difference between the current round and the previous round. In the case of N different use cases, a weighted sum of all the losses can be calculated as 1/(N+1), where the "1" in the denominator is contributed by the regularization loss. For example, if there are 2 use cases, the sum would be given as shown in Equation [3]:

Sum = 0.33* MSE_( prediction, l)+0.33*MSE_( prediction, 2)+0.33*MSE_(GenericWeightLoss, n) [3]

[0165] The weighted sum is the loss value of the overall ML model, consisting of the generic ML model and all tail ML models involved in training. The backward propagation is computed based on this loss value. The method then performs backwards propagation at the use case ML models and the generic ML model 22, and updates all models.

[0166] The steps of the model generalization phase are then repeated until ML model convergence.

[0167] In some embodiments, the regularization of the loss can also be made per use case for every round, n such as shown in Equation [4]:

[0168] Training results

[0169] Results of the methods described above were obtained over 30 experiment iterations. [0170] In the experiment use cases UC 0, UC 1 and UC 2 had the tasks of estimating the number of scheduled activities in the UE in the downlink (pmSchedActivityUeDI), the maximum number of RRC users (pmRRCConnMax), and physical resource block utilization in the downlink (prbUtilDownlink), respectively.

[0171] Figures 10A, 10B and 10C summarize a comparison of standalone split learning training versus the Methods 1, 2, 3 and 4 for the three use cases (UCO, UC1, and UC2, respectively). As compared to the standalone SL training scenarios, two out of three UCs benefit from collaborative SL training methodology. The convergence rate (the required number of SL rounds at the use case ML model) to reach high accuracy values is much higher (i.e., the time to convergence is less) for UC1, indicating a clear benefit of the proposed split learning methods in the generic ML model training. In addition, Methods 2, 3, and 4 address the catastrophic interference (high oscillations during training) as compared to Method 1. Moreover, in UC 2, all proposed methods already reach a high accuracy level (e.g., 0.6) at the 25th round, while the standalone SL case the accuracy starts increasing to above 0 after the 50th round.

[0172] NWDAF implementation

[0173] The training of the joint ML model with the participation of NWDAF and other NWDAFs may be performed in a sequential or in a hybrid SL setting. The generic ML model contains the first K layers of an overall NN model, and each use case ML model contains the tail portion, i.e., the last L layers of the NN model, where each use case ML model is specific/personalized to particular use cases. The generic ML model is jointly trained with different use case ML models (i.e., NWDAF clients). Figures 11A and 11B illustrate a procedure for the training of a generic ML model via hybrid split learning with UC classifier using a combination of sequential and parallel training with gradient reversal (Method 3).

[0174] Referring to Figure 11A, the initial training phases of use case ML model 1 and use case ML model 2 are illustrated in steps 1-9 and steps 10-18, respectively.

[0175] For example, in step 1, the NWDAF sends a training notification to client 1. At step 2, the NWDAF performs a forward pass and obtains the activations X', which are sent to client 1 at step 3. At step 4, client 1 performs a forward pass to obtain outputs y. [0176] Client 1 then calculates the error (y' - y) (where y' is the estimated value obtained at the end of each use case ML model, and y is the actual (ground truth) value) and calculates gradients by performing a backward pass using the obtained gradients at step 5.

[0177] At step 6, client 1 updates its model weights, and at step 7 sends the gradients to the NWDAF. The NWDAF performs a backward pass on the generic ML model at step 8 and updates the weights of the generic ML model at step 9.

[0178] Similar operations are then performed with client 2 at steps 10 to 18.

[0179] During the training of two use case ML models, the generic ML model (trained in a NWDAF with generic split model training capability) is sequentially updated. After training use case ML model 1 with client 1, the activations X' generated on the last forward pass of training are recorded. Then, after training use case ML model 2 with client 2, the activations X' generated on the last forward-pass of training are recorded. These two activations are sent to the use case classifier model (which reside within the NWDAF) together with the labels indicating which activations was intended to be sent to which client, i.e., use case ML model. This is illustrated in steps 19 and 20.

[0180] The UC classifier then trains its own ML model until convergence, so that it can classify activations. This involves performing a forward pass using the received activations (step 21). The output of the ML model, which represents the probability that an activation corresponds to a given use case, is obtained from the forward pass. Next, the error is computed at the UC classifier (step 22), and gradients for the UC classifier model are obtained at step 23. These steps may be iterated multiple times until the UC classifier model converges (e.g., when the UC classifier has the ability to classify the activations to the use case names with an acceptable accuracy).

[0181] Upon completion of training, the UC classifier sends a notification to the server indicating that the generalized training will start (Steps 24-26). The generic ML model performs a forward pass (step 26) and performs a few more rounds of training with use case ML model 1 (steps 27-39). At step 39, the generic ML model obtains and records the latest gradients from the last round of training with use case ML model 1. [0182] Referring to Figure 11B, the generic ML model performs a few rounds of training with the use case ML model 2 (40-53) and records the gradients received at the last round.

[0183] The last activations of the generic ML model that were recorded during the training with use case ML models 1 and 2 are then sent to the UC classifier model with the corresponding labels in steps 55 and 56. The use case classifier model trains for a few more rounds alone (steps 57-59).

[0184] The UC classifier model then reverses the sign of the gradients and sends the reversed gradients to the generic ML model in steps 60 and 61. In steps 62-64, the generic ML model aggregates the gradients and performs a backward pass followed by a weight update.

[0185] Steps 1-64 are repeated until convergence, with the exception that the initial model training phases in steps 1-18 consist of small number of rounds. Step 65 ends the training, and in step 66 the statistics of the model is registered at the MTLF.

[0186] In some embodiments, there may be many different regional NWDAF's. In that case, the generic head node can be federated between regional NWDAF head nodes. In addition, the use case tail portion of the models can be federated in between use case specific federations.

[0187] Once the generic ML model is trained it may be sent to an AF (e.g., to a specific UC model) or to the NWDAF. If the model is sent to the NWDAF, then there may be communications between MTLF for composing models. In the current 3GPP specification (3GPP rel. 17), an address to the generic ML model contained in the MTLF (used in the generic ML model training) is used by the clients to retrieve the ML model. The clients then retrieve the model and compose the individually trained ML models themselves.

[0188] Operations of systems/methods for performing split learning of a neural network model are illustrated in Figure 12. As shown therein, the operations include providing a first ML model that receives a set of input values (X) and generates a set of output activations (X') in response to the input values (block 1202). The systems/methods provide the output activations as inputs to a plurality of second ML models (block 1204). Each of the second ML models generates a different output (y) based on the set of output activations provided by the first ML model. The systems/methods jointly train the first ML model and the plurality of second ML models using the set of output activations (block 1206). During the joint training, the systems/methods attempt to generalize model weights of the first ML model such that the first ML model outputs representable and generalized features in the form of activations for different tasks implemented at the plurality of second ML models.

[0189] The first ML model may correspond to a generic head model of a ML model, and the second ML models may correspond to the UC-specific tail ML models of the ML model.

[0190] Referring to Figure 13, some embodiments provide a method performed by a client network function in a communication network for training a split ML model, wherein the split ML model comprises a second (tail) ML model of the split ML model. The method includes transmitting a discovery request to a discovery function in the communication network (block 1302). The discovery request requests an identity of a data analytics function in the communication network that manages first ML models that can be used to provide input to the second ML model. The client network function receives from the data analytics function, information indicating whether a first ML model managed by the data analytics function is suitable for training the second ML model (block 1304), and initiates split training with the data analytics function of a ML model including the second ML model and the first ML model based on the information received from the data analytics function (block 1306).

[0191] The first ML model may correspond to a generic head model of a ML model, and the second ML model may correspond to a UC-specific tail ML model.

[0192] The discovery request may include analytics metadata relating to the second ML model.

[0193] The information received from the data analytics function may include a training relation graph that indicates suitability of the first ML model for training with the second ML model and/or an estimated gain associated with training the second ML model with the first ML model.

[0194] The communication network may be a core network of a wireless communication system, and the data analytics may be comprises a NWDAF of the core network. The client network function may also be a NWDAF or another type of network function. [0195] The data analytics function may host a MTLF that performs joint training of the ML model.

[0196] Split training of the ML model may include joint training of the first ML model with a plurality of split training clients, each of which trains an associated second ML model. In particular, the first ML model may receive a set of input values (X) and generate a set of output activations (X') in response to the input values, wherein the output activations are provided as inputs to the second ML models associated with the split training clients. Each of the second ML models may generate a different output (y) based on the output activations generated by the first ML model. The joint training may include jointly training the first ML model with the split training clients using the set of output activations.

[0197] Figure 14A is a block diagram illustrating elements of a data analytics function 100 according to some embodiments. The data analytics function 100 may implement, for example, an NWDAF client that manages a UC tail ML model or an NWDAF server that manages a first ML model as illustrated, for example, in Figure 5.

[0198] The data analytics function 100 may be provided by, e.g., a device in the cloud running software on cloud computing hardware; or a software function/service governing or controlling a wireless communication network. That is, the device may be implemented as part of a communications system, or on a device as a separate functionality/service hosted in the cloud. The device also may be provided as a standalone software for managing a wireless communication network; and the device may be in a deployment that may include virtual or cloud-based network functions (VNFs or CNFs) and even physical network functions (PNFs). The cloud may be public, private (e.g., on premises or hosted), or hybrid.

[0199] As shown, the data analytics function 100 may include transceiver circuitry 1401 (e.g., RF transceiver circuitry) including a transmitter and a receiver configured to provide uplink and downlink radio communications with devices (e.g., a controller for automatic execution of actuations). The device may include network interface circuitry 1408 (also referred to as a network interface,) configured to provide communications with other devices (e.g., a controller for automatic execution of an actuation). The device may also include processing circuitry 1403 (also referred to as a processor) coupled to the transceiver circuitry, memory circuitry 1405 (also referred to as memory) coupled to the processing circuitry.

[0200] As discussed herein, operations of the data analytics function 100 may be performed by processing circuitry 1403, network interface 1408, and/or transceiver 1401. For example, processing circuitry 1403 may control the data analytics function 100 to perform operations according to embodiments disclosed herein. Processing circuitry 1403 also may control transceiver 1401 to transmit downlink communications through transceiver 1401 over a radio interface to one or more devices and/or to receive uplink communications through transceiver 1401 from one or more devices over a radio interface. Similarly, processing circuitry 1403 may control network interface 1408 to transmit communications through network interface 1408 to one or more devices and/or to receive communications through network interface from one or more devices. Moreover, modules may be stored in memory 1405, and these modules may provide instructions so that when instructions of a module are executed by processing circuitry 1403, processing circuitry 1403 performs respective operations (e.g., operations discussed below with respect to example embodiments relating to devices). According to some embodiments, the data analytics function 100 and/or an element(s)/function(s) thereof may be embodied as a virtual device/devices and/or a virtual machine/machines.

[0201] According to some other embodiments, a device may be implemented without a transceiver. In such embodiments, transmission to a wireless device may be initiated by the data analytics function 100 so that transmission to the wireless device is provided through a device including a transceiver (e.g., through a base-station). According to embodiments where the device includes a transceiver, initiating transmission may include transmitting through the transceiver.

[0202] Figure 14B illustrates various functional modules stored in the memory 1405 of the data analytics function 100. In particular, the memory 1405 may include a joint training model 1412 for conducting operations involving joint training of a split neural network model as either a client or a server. The memory 1405 may further include a generic head training module 1414 for performing forward and backward training passes on a generic head model and/or a tail ML model training module 1418 for jointly training a tail ML model with a generic head model. The memory 1405 may further include a UC classifier module for training a UC classifier model as illustrated, for example, in Figure 9.

[0203] Other modules may also be provided in the memory 1405 to effect the operations described above.

[0204] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and/or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and/or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and/or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0205] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and/or by end users and a wireless network generally.

Claims

CLAIMS:

1. A computer-implemented method of training a machine learning (ML) model using split learning, comprising: providing (1202) a first ML model (22) that receives a set of input values (X) and generates a set of output activations (X') in response to the input values; providing (1204) the output activations as inputs to a plurality of second ML models (24), wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model; jointly training (1206) the first ML model and the plurality of second ML models using the set of output activations, wherein jointly training the first ML model and the plurality of second ML models comprises simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training; and wherein during the joint training, model weights of the first ML model are generalized such that the first ML model outputs in the form of activations that represent representable features for different tasks implemented at the plurality of second ML models.

2. The method of Claim 1, further comprising: generating, at the plurality of second ML models, respective outputs related to the different tasks implemented at the plurality of second ML models.

3. The method of Claim 2, wherein the tasks implemented at the plurality of second ML models comprise generating predictions of network performance measurements and/or network events in a communication network.

4. The method of Claim 3, wherein the network performance measurements and/or network events comprise one or more of throughput, handovers, call drops, physical resource block utilization, channel quality degradation, and latency in the communication network.

5. The method of any previous Claim, wherein simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training comprises aggregating gradients from the plurality of second ML models to provide aggregated gradients and performing a backward pass on the first ML model using the aggregated gradients.

6. The method of any previous Claim, wherein the first ML model comprises a first neural network comprising a first plurality of layers, and wherein the second ML models comprise respective second neural networks.

7. The method of any previous Claim, wherein the output activations comprise outputs generated at an output layer of the first ML model.

8. The method of any previous Claim, wherein each of the second ML models is configured to perform a different estimation function.

9. The method of Claim 8, wherein the different functions comprise estimation functions or prediction functions.

10. The method of Claim 9, wherein the input values comprise performance monitor counters for a communication network, and wherein the estimation functions perform one or more of handover, throughput, call drop, physical resource block utilization, channel quality degradation, and latency in the communication network.

11. The method of any previous Claim, wherein jointly training the first ML model and the set of second ML models comprises performing initial training on the first ML model.

12. The method of Claim 11, wherein the initial training is performed jointly with at least one of the second ML models.

13. The method of Claim 12, wherein the initial training is performed together with a set of the second ML models in sequential phases.

14. The method of Claim 13, wherein the first ML model is trained together with each of the set of second ML models as an overall ML model in sequential phases until convergence of the overall ML model in each phase.

15. The method of any of Claims 12 to 14, wherein jointly training the first ML model and the set of second ML models comprises training the first ML model in an initial training phase followed by training the first ML model and the set of second ML models in a model generalization phase.

16. The method of Claim 15, wherein training the first ML model and the plurality of second ML models in the initial training phase comprises training the first ML model together with each of the second ML models in sequential phases until convergence.

17. The method of Claim 15 or 16, wherein training the first ML model and the plurality of second ML models in the model generalization phase comprises: for each of the plurality of second ML models, sequentially training the first ML model and the second ML model together for one round, performing a forward pass on the first ML model, recording activations generated by the first ML model, and obtaining a loss of the second ML model; aggregating gradients of the second ML models; performing backpropagation on the first ML model using the aggregated gradients; and updating the first ML model and the second ML models based on results of the backpropagation.

18. The method of Claim 17, wherein the operations of sequentially training the first ML model and each of the plurality of second ML models for one round and updating the first ML model and the second ML models after each round is performed until convergence of the first ML model and the second ML models.

19. The method of any of Claims 12 to 18, further comprising: providing the activations generated by the first ML model after training the first ML in the initial training phase to a use case classifier model; and training the use case classifier model to predict activations belonging to each of the second ML models.

20. The method of Claim 19, further comprising, in the model generalization phase, at each round of joint training of the first ML and one of the second ML models, training the use case classifier for one round with activations generated at the round of joint training, computing a cross entropy loss at an output of the use case classifier model, obtaining gradients of the use case classifier model, reversing signs of the gradients of the use case classifier model to obtain sign-reversed gradients, providing the sign-reversed gradients of the use case classifier model to the first ML model, and aggregating the sign-reversed gradients of the use case classifier model with the gradients of the second ML models before performing backpropagation on the first ML model.

21. The method of any of Claims 12 to 20, further comprising: calculating a loss function at each backpropagation operation in the generalized training round; and regularizing losses of the second ML models based on the loss function.

22. The method of Claim 21, wherein the loss function computes a difference between first ML model weights between two consecutive split learning rounds.

23. The method of Claim 22, wherein the loss function comprises a first component that computes a difference between first ML model weights between two consecutive split learning rounds and a second component that computes a loss of a second ML model, wherein the first component and the second component are combined to calculated losses when performing backpropagation of the models.

24. The method of Claim 23, wherein the first component and the second component comprise mean square error loss functions.

25. The method of Claim 24, wherein the first component is given by the equation:

MSE_GenericWeightLoss, n ~ Q ~ Q ) wherein Qⁿ is a weight matrix of the first ML model at training round n, and wherein the second component is given by the equation: wherein i is a sample index and y¹ is an actual output of the second ML model for which the loss function is evaluated, and y¹' is an estimated output of the second ML model.

26. A computer-implemented method performed by a client network function in a communication network for training a split machine learning, ML, model, wherein the split ML model comprises a second ML model of the split ML model, the method comprising: transmitting (1302) a discovery request to a discovery function in the communication network, wherein the discovery request requests an identity of a data analytics function in the communication network that manages first models; receiving (1304), from the data analytics function, information indicating whether a first model managed by the data analytics function is suitable for training the second ML model; and initiating (1306) split training with the data analytics function of a ML model including the second ML model and the first ML model based on the information received from the data analytics function.

27. The method of Claim 26, wherein the discovery request comprises analytics metadata relating to the second ML model.

28. The method of Claim 26 or T1 , wherein the information received from the data analytics function comprises a training relation graph that indicates suitability of the first ML model for training with the second ML model.

29. The method of any of Claims 26 to 28, wherein the information received from the data analytics function comprises an estimated gain associated with training the second ML model with the first ML model.

30. The method of any of Claims 26 to 29, wherein the communication network comprises a core network of a wireless communication system, and wherein the data analytics function comprises a network data analytics function, NWDAF, of the core network.

31. The method of Claim 30, wherein the client network function comprises a NWDAF.

32. The method of Claim 30 or 31, wherein the data analytics function hosts a model training logical function that performs joint training of the first ML model and the second ML models.

33. The method of any of Claims 26 to 32, wherein the split training of the ML model comprises joint training of the first ML model with a plurality of split training clients, each of which trains an associated second ML model.

34. The method of Claim 33, wherein the first ML model receives a set of input values (X) and generates a set of output activations (X') in response to the input values, wherein the output activations are provided as inputs to the second ML models associated with the split training clients, wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model; and wherein joint training comprises jointly training the first ML model with the split training clients using the set of output activations.

35. A computing device, comprising: a processor circuit; and a memory coupled to the processor circuit, wherein the memory stores computer program instructions that, when executed by the processor circuit, cause the computing device to perform operations comprising: providing (1202) a first ML model (22) that receives a set of input values (X) and generates a set of output activations (X') in response to the input values; providing (1204) the output activations as inputs to a plurality of second ML models (24), wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model; jointly training (1206) the first ML model and the plurality of second ML models using the set of output activations, wherein jointly training the first ML model and the plurality of second ML models comprises simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training; and wherein during the joint training, model weights of the first ML model are generalized such that the first ML model outputs in the form of activations that represent representable features for different tasks implemented at the plurality of second ML models.

36. The computing device of Claim 35, wherein the program instructions configure the computing device to perform operations of any of Claims 2 to 34.

37. A computer program product comprising a computer-readable storage medium that stores computer program instructions that, when executed by a processor circuit of a computing device, cause the computing device to perform operations comprising: providing (1202) a first ML model (22) that receives a set of input values (X) and generates a set of output activations (X') in response to the input values; providing (1204) the output activations as inputs to a plurality of second ML models (24), wherein each of the second ML models generates a different output (y) based on the output activations generated by the first ML model; jointly training (1206) the first ML model and the plurality of second ML models using the set of output activations, wherein jointly training the first ML model and the plurality of second ML models comprises simultaneously training the first ML model and the plurality of second ML models for at least a portion of the joint training; and wherein during the joint training, model weights of the first ML model are generalized such that the first ML model outputs in the form of activations that represent representable features for different tasks implemented at the plurality of second ML models.

38. The computer program product of Claim 37, wherein the computer program instructions configure the computing device to perform operations of any of Claims 2 to 34.