WO2025201766A1

WO2025201766A1 - Apparatus, method and computer program

Info

Publication number: WO2025201766A1
Application number: PCT/EP2025/054829
Authority: WO
Inventors: Afef Feki; Sakira HASSAN; István Zsolt KOVÁCS
Original assignee: Nokia Technologies Oy
Current assignee: Nokia Technologies Oy
Priority date: 2024-03-28
Filing date: 2025-02-24
Publication date: 2025-10-02
Anticipated expiration: 2026-09-28
Also published as: GB2639969A; GB202404592D0

Abstract

There is provided an apparatus comprising means for performing receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured, determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

Description

Title

Apparatus, method and computer program

Field

The present application relates to a method, apparatus, system and computer program and in particular but not exclusively to user equipment (UE) machine learning (ML) update based on network side model delta threshold configuration.

Background

A communication system can be seen as a facility that enables communication sessions between two or more communication devices, or provides communication devices access to a network. A mobile or wireless communication network is one example of a communication network. A communication device may be provided with a service by an application server.

Such communication networks operate in according with standards such as those provided by 3GPP (Third Generation Partnership Project) or ETSI (European Telecommunications Standards Institute). Examples of standards are the so-called 4G (4th Generation), 5G (5th Generation) standards provided by 3GPP.

Summary

In a first aspect there is provided an apparatus comprising means for performing receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured, determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

If the performance information indicates a value above a first threshold, the updated model delta threshold value may be equal to a model delta value indicated by the model delta information and if the performance information indicates a value below the first threshold, the updated model delta threshold value may be equal to the minimum of a sum of the model delta threshold value and an offset and the model delta value indicated by the model delta information.

The apparatus may comprise means for performing receiving further information relating to the machine learning model.

The further information relating to the machine learning model may comprise at least one of the following: model delta information relating to an updated machine learning model, performance information relating to the updated machine learning model or the updated machine learning model.

The apparatus may comprise means for performing: determining to update the machine learning model based on the further information and providing an indication at least to the entity on which the machine learning model is configured to update the machine learning model.

The further information may comprise the updated machine learning model and comparing means for performing: determining to provide the updated machine learning model to at least one further entity on which the machine learning model is configured.

The apparatus may comprise means for receiving performance information and model delta information relating to the machine learning model from a plurality of entities on which the machine learning model is configured and performing: determining a confidence value for a correspondence between the received model delta information and performance.

The apparatus may comprise means for providing a configuration to the entity on which the machine learning model is configured to configure machine learning model delta parameters.

The entity on which the machine learning model is configured may comprise a user equipment or a network function.

In a second aspect there is provided an apparatus comprising means for determining to update a machine learning model configured at the apparatus, updating the machine learning model; determining model delta information based on the machine learning model and the updated machine learning model, determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value and providing the information relating to the machine learning model to the network entity.

The apparatus may comprise means for further determining performance information based on the machine learning model and the updated machine learning model, providing the model delta information and the performance information to a network entity and receiving an updated model delta threshold value from the network entity in response.

The information relating to the machine learning model may comprise at least one of the following: the model delta information, performance information relating to the machine learning model or the updated machine learning model.

The model delta information may be determined based on at least a difference between weights of the machine learning model and weights of the updated machine learning model.

The model delta information may indicate at least one of a global machine learning delta value and a vector machine learning delta value.

In a third aspect there is provided a method comprising performing receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured, determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

The method may comprise performing receiving further information relating to the machine learning model. The further information relating to the machine learning model may comprise at least one of the following: model delta information relating to an updated machine learning model, performance information relating to the updated machine learning model or the updated machine learning model.

The method may comprise performing: determining to update the machine learning model based on the further information and providing an indication at least to the entity on which the machine learning model is configured to update the machine learning model.

The method may comprise receiving performance information and model delta information relating to the machine learning model from a plurality of entities on which the machine learning model is configured and performing: determining a confidence value for a correspondence between the received model delta information and performance.

The method may comprise providing a configuration to the entity on which the machine learning model is configured to configure machine learning model delta parameters.

In a fourth aspect there is provided a method comprising determining to update a machine learning model configured at the apparatus, updating the machine learning model, determining model delta information based on the machine learning model and the updated machine learning model, determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value and providing the information relating to the machine learning model to the network entity.

The method may comprise further determining performance information based on the machine learning model and the updated machine learning model, providing the model delta information and the performance information to a network entity and receiving an updated model delta threshold value from the network entity in response. The information relating to the machine learning model may comprise at least one of the following: the model delta information, performance information relating to the machine learning model or the updated machine learning model.

In a fifth aspect there is provided an apparatus comprising at least one processor, and at least one memory storing instructions which, when executed by the processor, cause the apparatus at least to perform the method according to the third or fourth aspect.

In a sixth aspect there is provided a computer readable medium comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the method according to the third or fourth aspect.

In a seventh aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the method according to the third or fourth aspects.

In the above, many different embodiments have been described. It should be appreciated that further embodiments may be provided by the combination of any two or more of the embodiments described above.

Description of Figures

Embodiments will now be described, by way of example only, with reference to the accompanying Figures in which:

Figure 1 shows a schematic diagram of an example 5GS communication system;

Figure 2 shows a schematic diagram of an example mobile communication device;

Figure 3 shows a schematic diagram of an example control apparatus; Figure 4 shows an example call flow for model download and LCM activities for a ML enabled feature;

Figure 5 shows a flowchart of a method according to an example embodiment;

Figure 6 shows a flowchart of a method according to an example embodiment;

Figure 7 shows a call flow according to an example embodiment;

Figure 8 shows a flow diagram of adapted model data threshold selection and model delta use;

Figure 9 shows a flow diagram of adapted model data threshold selection.

Detailed description

Before explaining in detail the examples, certain general principles of a wireless communication system and mobile communication devices are briefly explained with reference to Figure 1 , Figure 2 and Figure 3 to assist in understanding the technology underlying the described examples.

An example of a suitable communications system is the 5G or NR concept. Network architecture in NR may be similar to that of LTE-advanced. Base stations of NR systems may be known as next generation NodeBs (gNBs). Changes to the network architecture may depend on the need to support various radio technologies and finer Quality of Service (QoS) support, and some on-demand requirements for e.g. QoS levels to support Quality of Experience (QoE) for a user. Also network aware services and applications, and service and application aware networks may bring changes to the architecture. Those are related to Information Centric Network (ICN) and User-Centric Content Delivery Network (UC-CDN) approaches. NR may use Multiple Input - Multiple Output (MIMO) antennas, many more base stations or nodes than the LTE (a so-called small cell concept), including macro sites operating in co-operation with smaller stations and perhaps also employing a variety of radio technologies for better coverage and enhanced data rates. Future networks may utilise network functions virtualization (NFV) which is a network architecture concept that proposes virtualizing network node functions into “building blocks” or entities that may be operationally connected or linked together to provide services. A virtualized network function (VNF) may comprise one or more virtual machines running computer program codes using standard or general type servers instead of customized hardware. Deployments may be cloud-native network function (CNF) based, where network functions comprise one or more pods. Cloud computing or data storage may also be utilized. In radio communications this may mean node operations are to be carried out, at least partly, in a server, host or node operationally coupled to a remote radio head. It is also possible that node operations will be distributed among a plurality of servers, nodes or hosts. It should also be understood that the distribution of labour between core network operations and base station operations may differ from that of LTE or may even be non-existent.

Figure 1 shows a schematic representation of a 5G system (5GS) 100. The 5GS may comprise a user equipment (UE) 102 (which may also be referred to as a communication device or a terminal), a 5G radio access network (5GRAN) 104, a 5G core network (5GCN) 106, one or more internal or external application functions (AF) 108 and one or more data networks (DN) 110.

An example 5G core network (CN) comprises functional entities. The 5GCN 106 may comprise one or more Access and mobility Management Functions (AMF) 112, one or more session management functions (SMF) 114, an authentication server function (ALISF) 116, a Unified Data Management (UDM) 118, one or more user plane functions (UPF) 120, a Unified Data Repository (UDR) 122 and/or a Network Exposure Function (NEF) 124. The UPF is controlled by the SMF (Session Management Function) that receives policies from a PCF (Policy Control Function).

The CN may be connected to a UE via the Radio Access Network (RAN) or through fixed access via a non-3GPP Interworking Function (N3IWF). The 5GRAN may comprise one or more gNodeB (gNB) Distributed Unit (DU) functions connected to one or more gNodeB (gNB) Centralized Unit (CU) functions. The RAN may comprise one or more access nodes.

A User Plane Function (UPF) referred to as PDU Session Anchor (PSA) may be responsible for forwarding frames back and forth between the DN and the tunnels established over the 5G towards the UE(s) exchanging traffic with the DN. A possible mobile communication device will now be described in more detail with reference to Figure 2 showing a schematic, partially sectioned view of a communication device 200. Such a communication device is often referred to as user equipment (UE) or terminal. An appropriate mobile communication device may be provided by any device capable of sending and receiving radio signals. Non-limiting examples comprise a mobile station (MS) or mobile device such as a mobile phone or what is known as a ’smart phone’, a computer provided with a wireless interface card or other wireless interface facility (e.g., USB dongle), personal data assistant (PDA) or a tablet provided with wireless communication capabilities, voice over IP (VoIP) phones, portable computers, desktop computer, image capture terminal devices such as digital cameras, gaming terminal devices, music storage and playback appliances, vehiclemounted wireless terminal devices, wireless endpoints, mobile stations, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart devices, wireless customerpremises equipment (CPE), or any combinations of these or the like. A mobile communication device may provide, for example, communication of data for carrying communications such as voice, electronic mail (email), text message, multimedia and so on. Users may thus be offered and provided numerous services via their communication devices. Non-limiting examples of these services comprise two-way or multi-way calls, data communication or multimedia services or simply an access to a data communications network system, such as the Internet. Users may also be provided broadcast or multicast data. Non-limiting examples of the content comprise downloads, television and radio programs, videos, advertisements, various alerts, and other information.

A mobile device is typically provided with at least one data processing entity 201 , at least one memory 202 and other possible components 203 for use in software and hardware aided execution of tasks it is designed to perform, including control of access to and communications with access systems and other communication devices. The data processing, storage and other relevant components can be provided on an appropriate circuit board and/or in chipsets. This feature is denoted by reference 204. The user may control the operation of the mobile device by means of a suitable user interface such as key pad 205, voice commands, touch sensitive screen or pad, combinations thereof or the like. A display 208, a speaker and a microphone can be also provided. Furthermore, a mobile communication device may comprise appropriate connectors (either wired or wireless) to other devices and/or for connecting external accessories, for example hands-free equipment, thereto.

The mobile device 200 may receive signals over an air or radio interface 207 via appropriate apparatus for receiving and may transmit signals via appropriate apparatus for transmitting radio signals. In Figure 2 transceiver apparatus is designated schematically by block 206. The transceiver apparatus 206 may be provided for example by means of a radio part and associated antenna arrangement. The antenna arrangement may be arranged internally or externally to the mobile device.

Figure 3 shows an example of a control apparatus 300 for a communication system, for example to be coupled to and/or for controlling a station of an access system, such as a RAN node, e.g. a base station, eNB or gNB, a relay node or a core network node such as an MME or Serving Gateway (S-GW) or Packet Data Network Gateway (P-GW), or a core network function such as AMF/SMF, or a server or host. The method may be implemented in a single control apparatus or across more than one control apparatus. The control apparatus may be integrated with or external to a node or module of a core network or RAN. In some embodiments, base stations comprise a separate control apparatus unit or module. In other embodiments, the control apparatus can be another network element such as a radio network controller or a spectrum controller. In some embodiments, each base station may have such a control apparatus as well as a control apparatus being provided in a radio network controller. The control apparatus 300 can be arranged to provide control on communications in the service area of the system. The control apparatus 300 comprises at least one memory 301 , at least one data processing unit 302, 303 and an input/output interface 304. Via the interface the control apparatus can be coupled to a receiver and a transmitter of the base station. The receiver and/or the transmitter may be implemented as a radio front end or a remote radio head.

Machine learning (ML) is a branch of Artificial Intelligence (Al) which uses algorithms which learn from data and make predictions from inputs without explicit programming.

Machine learning models may include neural networks. Neural networks are formed of nodes or “artificial neurons” which are connected by edges. The output of each neuron is computed by some non-linear function of the sum of its inputs, called the activation function and sent to other connected neurons. Neurons and edges typically have a weight that adjusts as learning proceeds. Neurons may be aggregated into layers. Different layers may perform different transformations on their inputs.

AI/ML based solutions are foreseen to apply for several use cases in radio access networks. Initial cases have been identified in the 3GPP standard in RAN1 and RAN3 which include energy saving, CSI compression, CSI feedback enhancement, positioning accuracy enhancement and beam management as examples. Enabling AI/ML solutions in an entity (e.g., gNB, UE, CN, OAM, LMF) of a communications network may require transferring (or delivering) the AI/ML model from one entity to another, destination, entity for inference/prediction using the model in the destination entity. The signaling and protocol for the transfer of an ML model is under discussion.

Some possible solutions may include a gNB transferring AI/ML model(s) to UE via CP (e.g., RRC signalling) or UP data, CN (except LMF) transferring AI/ML model(s) to UE via CP (e.g., NAS signalling) or UP data, LMF transferring AI/ML model(s) to UE via CP (e.g., LPP signalling) or UP data or a server (e.g. OAM, OTT) transferring AI/ML model(s) to UE (e.g., transparent to 3GPP).

The pros and cons of CP and UP based solution for model transfer are to be studied, but there is no consensus on the model delta configuration.

Figure 4 illustrates a call flow for model download and life cycle management (LCM) activities for an ML enabled feature. The model is stored at a network in step 1. In step 2, the UE requests Model A. In step 3, the network provides Model A to the UE in a download. In step 4, LCM operations (e.g., inference, monitoring, re-training etc) related to model A are performed. Once the model is downloaded to the UE, the UE may receive additional information for other LCM operations for any ML enabled feature. The additional information may include, for example, configuration of data collection, configuration of model training, configuration of the model usage, additional hardware/software (deployment) configuration information to run the model, configuration of generating the measurement reports or feedback to the NW and so on.

After the model is deployed in the UE device, it is yet to be clarified that the model being used (or “active”) in the UE requires no update at all or whether this model can be changed (model switching) or updated (model retraining) over the course of the time.

In order to ensure the desired ML performance in all conditions and environments variations, it may be necessary to proceed with ML model update once the model is deployed to cope with environment/context changes. Due to, for example, a change in the radio environment, data that were used to train a UE ML model may become obsolete and an initial active model may require update/tuning. After training/retraining/tuning/applying transfer learning, at least some of the following information may be stored regarding the model regardless of any implementation specific platform (e.g., tensorflow, pytorch, mxnet): the weights of the model (for inference), the state of the model, training configuration (eg., optimizer, losses, metrics, learnable parameters), checkpoints (for resume training/retraining) or the architecture of the model.

This information may be used to configure the ML model.

Generally, the model update refers to a re-estimation of the model, or parts of the model, parameters realized through retraining or refinement. In some cases, the model update is performed by a different entity than the one which is realizing the inference. Thus, the updated model may need to be transferred to the UE. Conversely, the model may be updated at the UE and may need to be transferred to the NW (or the NW may need to be informed of the updated). However, if the differences between the existing model and the updated one are insignificant, the transfer may be as unnecessary and wasteful of radio resources. Even in the case where the model is not transferred, it is valuable to the network to assess to which extent the model update is significant with regards to the achieved performance, as part of ML monitoring process.

In terms of 3GPP specification, UE model updates may impact and/or change the supported ML-enabled Functionality/Feature i.e., the network may take specific action before/during/after the UE ML model updates. Considering the mobility of the UE, the UE may need to adjust the same model in the different environment (e.g., cell change) for the same functionality/ML enabled feature. Therefore, for certain use cases and scenarios where the supported ML- enabled Functionality/feature is impacted in terms of required network configuration, reporting configuration, signaling, etc., it is important to define a network-controlled UE ML update mechanism as well as how to perform it.

There is a question of how to monitor ML model running at UE side without explicit model transfer when the UE model is updated.

Figure 5 shows a flowchart of a method according to an example embodiment. The method may be performed at/on/by an apparatus, such as a network entity. The entity on which the machine learning model is configured may comprise a user equipment or a network function or node such as, but not limited to gNB, CN, GAM or LMF. In 501 the method comprises receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured.

In 502 the method comprises determining to update a model delta threshold value based on at least the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model. The method includes updating model delta threshold value based on at least the received performance information and model delta information.

In 503, the method comprises providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured. Providing an indication of the updated model threshold to the entity may be considered as sharing the updated model threshold with the entity.

Figure 6 shows a flowchart of a method according to an example embodiment. The method may be performed at/on/by an apparatus, such as a UE as described with reference to Figure 2 or a network function or node such as, but not limited to gNB, CN, OAM or LMF.

In 601 , the method comprises determining to update a machine learning model configured at the apparatus.

In 602, the method comprises updating the machine learning model.

In 603, the method comprises determining model delta information based on the machine learning model and the updated machine learning model.

In 604, the method comprises determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value.

In 605, the method comprises providing the information relating to the machine learning model to the network entity. A method as described with reference to Figures 5 and 6 may provide a framework for ML model monitoring based on model delta information and its correspondence with the achievable performance for a given ML functionality. Considering two ML models, the model delta refers to the gap between old and new models.

Method as described above with reference to Figures 5 and 6 may provide an estimate of the model delta (i.e., a difference or gap) between the models (e.g., an initial model and the updated model). The UE may evaluate to which extent this model delta is significant from a performance perspective at UE side. Thereafter, this model delta (which is an example of model delta information) can be used at the network side to track the performance of the considered ML functionality and the extent to which a modification/update has been made and decide monitoring actions accordingly. The benefit of such a method may also be to facilitate recovery and fallback to older version of the model based on the model delta information.

Depending on the considered use case and ML model architecture, the model delta information may be translated to a different impact on the model performance, for example a model delta value of 10% may relate to models quite close in terms of performance and accuracy for one functionality whereas, for another functionality, a model delta value of 10% may be high and the performance for the two considered ML models is quite diverging.

Performance information may indicate achievable performance and may comprise accuracy information, e.g., an indication of accuracy, or other performance related information. Performance information may relate to one or more performance criteria.

The method may involve estimating a model delta threshold tailored to a given ML functionality and compliant with the supported model delta configuration.

The method may provide a framework (including a procedure and corresponding signalling) to perform ML Model monitoring based on model delta information, and, in some embodiments, the corresponding achievable performance. The method provides an estimate of a model delta threshold tailored to a given ML functionality and compliant with the supported model delta configuration: the model delta should translate this difference between the two models, either through global ML model delta (GMLD) value or vector ML model delta (VMLD).

Model delta information may be determined at least based on a difference between weights of the machine learning model and weights of the updated machine learning model. The model delta information may indicate at least one of a global machine learning delta value and a vector machine learning delta value.

A GMLD value may be defined as a unique value such as the empirical CDF at selected threshold is in the 95 percentile or also mean squared error (MSE) (between the total weights of the two models).

VMLD may be defined as a vector providing a more detailed view on the model delta information. In fact, VMLD may translate the difference (as calculated in the GMLD case) but for each layer. In this case it may be needed to have an indication of the layer index by a value (0, 1 , 2) or name (Dense_1 , Conv_1).

At a network side, there is an adaptive selection of the configured ML Model size threshold tailored to ML functionality and compliant with the supported model delta configuration (supported by the UE and which could be GMLD or VMLD) which translates the correspondence between the model delta information and the achievable model performance.

An aggregation at the network side of the UE reported GMLD/VLMD values, among multiple selected UEs, may be used to derive the confidence on the correspondence between the model/functionality performance and the GMLD/VLMD information.

At the network side, model and/or functionality performance monitoring may be provided using model delta information with corresponding achievable performance and relying on UEs reporting of the change in the model performance (e.g. accuracy). One advantage is to characterize and quantify the model updates at UE side accounting on the achieved functionality performance and without requiring full model information (through model transfer).

Signalling enhancements to collect the necessary information for the model delta threshold adaptive selection and enable model monitoring using model delta information may be provided.

Figure 7 shows an example signalling diagram according to an example embodiment. Figure 7 depicts the different exchanges between a UE and a network to enable model delta-based monitoring at Network side with UE assistance. The UE is an example of an entity on which a machine learning model is configured. Other entities on which a ML model may be configured include network entities, for example, gNB, CN, OAM or LMF. In step 1 , UE initiates the use of model delta by requesting its configuration from network through machine learning delta configuration message (MLDC) message.

In Step 2, the network responds by indicating the selected MLDC policy. The MLDC policy is an example of a configuration to configure machine learning model delta parameters. Step 2 is an example of providing a configuration to the entity on which the machine learning model is configured to configure machine learning model delta parameters.

The MLDC policy may be provided through control plane via, for example, RRC, MAC CE, DCI. The MLDC policy may indicate the configuration for an optimization phase which targets the estimation of a model delta threshold and a monitoring phase which aims at monitoring the model performance using model delta information and the selected model delta threshold. Machine learning model delta parameters may include the type of model delta parameter (e.g., GMLD or VMLD)

In step 3, at the UE side, the model is updated. The model may be triggered by e.g., model performance degradation or detection of a change in the environment. This is an example of determining to update a machine learning model configured at the apparatus and updating the machine learning model.

Thereafter, in steps 4 to 9, the UE estimates the model delta based on the configuration set by the network (which may be either GMLD value or VMLD value). This is an example of determining model delta information based on a machine learning model and an updated machine learning model. The model delta or model delta value, 5, is an example of model delta information.

Steps 10 to 12 may define a first, or optimisation, phase in which the network selects a model delta threshold for the considered ML functionality.

In step 10, the UE sends the model accuracy information (after model update) and the corresponding model delta. The model accuracy information is an example of performance information based on the updated machine learning model and the corresponding model delta is an example of model delta information based on the machine learning model and the updated machine learning model. Step 10 is an example of providing the model delta information and the performance information to a network entity. At step 11 , the network selects the threshold value and indicates it to the UE in step 12. Steps 11 and 12 are an example of determining to update a model delta threshold value based on the received performance information and model delta information and share an updated model delta threshold with the entity and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

Steps 13 to 15 may define a second, or monitoring, phase. In step 13 the UE computes the model delta each time the model is updated and compares it with the threshold indicated by the network (in step 12). This is an example of a comparison between the model delta information and the model delta threshold value.

If the model delta is higher than a model delta threshold then the UE sends model information (in step 14). The model information is an example of information relating to the machine learning model. Step 14 is an example of determining to provide information relating to the machine learning model to the network entity based on a comparison between the model delta information and the model delta threshold value. Information relating to the machine learning model may include at least one of the following: model delta information, performance information relating to the machine learning model or the updated machine learning model.

In step 15, based on received information relating to the ML model (in this example model delta information) from at least one UE (the at least one UE following the model delta threshold condition), the network performs monitoring. This has the advantage of filtering the information relating to the ML model communicated towards the network to relevant information (in terms of performance) and allows thereafter the network to perform more efficient monitoring decisions.

In an example embodiment, to gather more samples with the mapping or correspondence between performance and model delta (e.g., between accuracy gap acc_gap and the model delta value 5), the network may request one more UEs to perform model update (e.g., re- training/refining) (for the same ML-enabled Feature/Functionality) when it is possible (e.g. in case of low traffic and sufficient battery level).

The first, or optimisation, phase may be referred to as an adaptive selection of the model delta threshold.

Model delta information may indicate the delta (or gap) between two ML models. The two ML models may be two neural networks with similar architecture but different weight values. The second model may be an updated version of a model which has been updated (refined/ retrained) at a UE side. The model delta information may comprise a model delta value, 5, (e.g., a GMLD or VLMD value).

Figure 8 depicts an example procedure highlighting the use of the model delta value and the use of the threshold value which is updated in a dynamic manner for a given ML functionality (e.g., for a use case such as ML based positioning).

In step 1 of figure 7, a ML model is updated (i.e., (re)trained or finetuned) upon predefined criteria such as a detection of change in the environment. This is an example of determining to update a machine learning model configured at the apparatus and updating the machine learning model;

In step 2, the model delta value 5 is calculated following a MLDC received from the network.

Given a preselected model delta threshold value, 5₀, the estimated 8 is compared to it. If it is larger, then the new model is considered to be different from the previous one and thereafter its related information (e.g., model delta and corresponding performance gap) should be shared with the network (as shown in step 6). This step is an example of determining to provide information relating to the machine learning model to the network entity based on a comparison between the model delta information and the model delta threshold value Otherwise, the model information is not shared with the network.

In the meantime, in step 3, the updated model accuracy is estimated which provides a mapping between the model performance as indicate by model accuracy/performance gap acc_aav and the model delta value 8, where: acc_gap = acc_new — acc_init with acc_new refers to the updated model accuracy and acc_init is the accuracy of the initial model. Performance information relating to the updated machine learning model may comprise acc_gap or acc_new .

In Step 4, the threshold value 8₀ is updated. This step is performed by the network accounting for the previously reported mapping accuracy gap acc_gap and the model delta value 8. This is an example of determining to update a model delta threshold value based on received performance information and model delta information. In an example embodiment, at the beginning of an optimisation phase, the network can select a low 5₀ which induces sharing the model information (e.g., model delta, performance or could the complete model to transfer) frequently each time it is updated. However, by gathering more knowledge about the relationship between model delta and achievable model accuracy, the network can perform more fitted adjustment of the model delta threshold 5₀ in accordance with the desired accuracy requirement for the ML functionality. Additionally, the gathered information from UEs on this specific model evolution (with model delta) may be used for model monitoring.

Note that the performance and accuracy show the extent to which the model is performing well. However, in case the model is updated this does not indicate to which extent the updated model is different from the previous one, and consequently how much ML model data needs to be transferred due to the update, with other entities. Thus, model delta threshold is used as a criterion to indicate whether it is necessary to share the model information with the other entities (e.g. model transfer to the gNB).

Figure 9 shows a flowchart which details example steps for the model delta threshold update considering the different cases. It corresponds to an example for step 4 (Model delta threshold selection) in Figure 8. However other implementations of step 4 may be used for the proposed framework.

If acc_gap\s above a threshold and 8 < 8₀ then 8₀ is updated to 8.

If accg_ap'\s below a threshold and 8 > 8₀ then 8₀ is updated using formula 2 described in the example embodiment below.

In an embodiment, if the performance information indicates a value above a first threshold, the updated model delta threshold value is equal to a model delta value indicated by the model delta information and if the performance information indicates a value below the first threshold, the updated model delta threshold value is equal to the minimum of a sum of the model delta threshold value and an offset and the model delta value indicated by the model delta information.

The following is an example of model delta threshold 8₀ selection.

Initially the model delta threshold is selected with initial low value: 8₀ = 5%. Here, the model delta between initial model and updated model is calculated in terms of percentage difference between the weights values as follows:

5 = with W_u is the vector of updated model weights and is the vector of initial model weights [Formula 1]

The initial model delta threshold value is selected low to ensure sharing the model information each time it is updated since at this stage there is no knowledge gathered yet.

A first UE1 updates its model, calculates the model delta as indicated by the network 8 = 10% (using formula 1) and the related performance in percentage as acc_gap where acc_uand acc_t refer respectively to the model accuracy of updated and initial models. In this example UE1 indicates performance gap = 3% (which is low). With this shared information and assuming preselected fixed value 8_update , the network updates the threshold as: [Formula 2]

In fact, in this case, the new model does not bring much performance improvement so the model delta threshold can be increased up to 8 but we can be more conservative by setting up fixed value of 8_update value e.g. 3%. Thus, this the model delta threshold becomes in this example 5₀ = 8%.

A second UE2 updates also the ML model and estimates the model delta threshold 8₂ = 20% (using formula 1) and the corresponding model accuracy gap to acc_gap = 18% . As the performance difference is judged in this case to be significant (e.g. higher than a preselected limit of 15%), we compare 8₂ to 8₀.

As 8₂ > 8₀, this means that in this case the model information will be shared and then 8₀ is kept as it is to 8%.

A third UE3 updates also the ML model and estimates the model delta threshold <5₃ = 15% (using formula 1) and the corresponding model accuracy gap to acc_gap = 10% . As the performance difference is judged in this case not to be significant (as it is lower than the preselected limit of 15%), then the model delta threshold should be updated in this case following formula 1 which brings 8₀ = min(8 + 3; 15) = 11% Note that in this example we consider the update of the model delta threshold realized based on each UE reporting. However, the procedure may be made more robust to account for the case where multiple UEs report multiple model delta values but varying performance gap. In this case, we suggest that the network does not proceed by the model delta threshold update immediately but rather collect the information from multiple UEs and aggregate it before proceeding with the threshold update (e.g. for two UEs with same 8 = 20 % but performance gap of 5% and 18% the update is not made until getting more reporting from other UEs corresponding to this specific model delta of 20%).

The selection of the model delta threshold is relevant to indicate the correspondence between the model update and achieved performance. The main advantage is to get a view on the model evolution I update without need for the whole model transfer which helps the network to take better decisions regarding model/functionality switching. The network may receive model information from UEs, each time it is relevant which is the case where model delta is higher than the selected model threshold value. In this way, the network can have a view on the different significant model updates (in terms of achievable performance gain). Thereafter relevant actions can be undertaken by the network within this monitoring procedure. As example, a model update reported to the network with high performance gain and high model delta value from several UEs may trigger the network to request the model change/update over all UEs.

Further information relating to the machine learning model may comprise at least one of the following model delta information relating to an updated machine learning model, performance information relating to the updated machine learning model or the updated machine learning model.

In an example embodiment, when the further information comprises the updated machine learning model, a method may comprise determining to provide the updated machine learning model to at least one further entity on which the machine learning model is configured.

In an example embodiment, a method as described with reference to Figure 5 may comprise receiving performance information and model delta information relating to the machine learning model from a plurality of entities on which the machine learning model is configured and performing: determining a confidence value for a correspondence between the received model delta information and performance. For example, as part of the monitoring process, the network may consolidate its view on correlation between the model delta and corresponding performance gap/gain (in case one or more UE reports erroneous information). In fact, a confidence level can be associated to the model delta related information through the examination of the variance between the values reported by different UEs (considering the same ML model I functionality). In this way the monitoring decisions can also account for possible erroneous reporting for some UEs.

In an embodiment, the network may determine to retrain the machine learning model based on the further information and provide an indication at least to the entity on which the machine learning model is configured to retrain the machine learning model.

Table 1 illustrates how actions are chosen at the network based on the model delta threshold provided to the UE.

Changing the whole architecture of the ML model is one solution if the model performance is significantly degraded. However if ML model performance is below expectation, the NW may first update the model (with recalculation of the weights but keeping the same structure). This is less costly and time consuming compared to whole ML model structure change (which requires search for a new structure and calculation of the related parameters from scratch). This is an example of determining to retrain the ML model based on the further information.

Different model update techniques exist while keeping the mode structure the same. Depending on the extent of these updates, and how they have been achieved (how long retraining, which input date was used, etc), the update results in different performance. Quantization and pruning may further impact the final model performance. With model performance monitoring discussed in RAN1 , the alignment with two different datasets for accuracy is critical. For example, UE gets model A from NW. UE may use dataset 1 to retrain model A to model AA. Now in performance monitoring, UE will send ||A -AA|| to NW. NW will then have no clue what is being changed in the UE only given a value or performance metric. However, if model delta is being sent by the UE, then NW knows which parameters causes the changes and it is not coming only from different dataset distributions. The above provides a framework to indicate not only how a model is changed by a given environment, but a NW also learns how it the model changed from additional model delta information accounting on its performance impact for the considered use case (in other words, ML functionality).

An apparatus may comprise means for receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured, determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

The apparatus may comprise a network entity, be the network entity or be comprised in the network entity or a chipset for performing at least some actions of/for the network entity.

An apparatus may comprise means for determining to update a machine learning model configured at the apparatus, updating the machine learning model, determining model delta information based on the machine learning model and the updated machine learning model, determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value and providing the information relating to the machine learning model to the network entity.

The apparatus may comprise a UE as described with reference to Figure 2 or a network function or node such as, but not limited to gNB, CN, OAM or LMF , be the user equipment, network function or node or be comprised in the user equipment, network function or node or a chipset for performing at least some actions of/for the user equipment, network function or node. In an embodiment, the apparatus comprising a network function refers to a device/apparatus which is configured to perform or performs at least part of functionalities of the network function or to a device/apparatus which comprises circuitry (e.g., chipset) configured to perform or performing at least part of functionalities of the network function.

It should be understood that the apparatuses may comprise or be coupled to other units or modules etc., such as radio parts or radio heads, used in or for transmission and/or reception. Although the apparatuses have been described as one entity, different modules and memory may be implemented in one or more physical or logical entities.

It is noted that whilst some embodiments have been described in relation to 5G networks, similar principles can be applied in relation to other networks and communication systems such as 6G networks or 5G-Advanced networks. Therefore, although certain embodiments were described above by way of example with reference to certain example architectures for wireless networks, technologies and standards, embodiments may be applied to any other suitable forms of communication systems than those illustrated and described herein.

It is also noted herein that while the above describes example embodiments, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.

As used herein, “at least one of the following: <a list of two or more elements>” and “at least one of <a list of two or more elements>” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

In general, the various embodiments may be implemented in hardware or special purpose circuitry, software, logic or any combination thereof. Some aspects of the disclosure may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto. While various aspects of the disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. As used in this application, the term “circuitry” may refer to one or more or all of the following:

(a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry) and

(b) combinations of hardware circuits and software, such as (as applicable):

(i) a combination of analog and/or digital hardware circuit(s) with software/firmware and

(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

The embodiments of this disclosure may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and/or macros, may be stored in any apparatus-readable data storage medium and they comprise program instructions to perform particular tasks. A computer program product may comprise one or more computerexecutable components which, when the program is run, are configured to carry out embodiments. The one or more computer-executable components may be at least one software code or portions of it.

Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD. The physical media is a non-transitory media. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e. , tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may comprise one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), FPGA, gate level circuits and processors based on multi core processor architecture, as non-limiting examples.

Embodiments of the disclosure may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

The scope of protection sought for various embodiments of the disclosure is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the disclosure.

The foregoing description has provided by way of non-limiting examples a full and informative description of the exemplary embodiment of this disclosure. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this disclosure will still fall within the scope of this invention as defined in the appended claims. Indeed, there is a further embodiment comprising a combination of one or more embodiments with any of the other embodiments previously discussed.

Claims

1. An apparatus comprising means for performing: receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured; determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model; and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

2. The apparatus according to claim 1, wherein, if the performance information indicates a value above a first threshold, the updated model delta threshold value is equal to a model delta value indicated by the model delta information and if the performance information indicates a value below the first threshold, the updated model delta threshold value is equal to the minimum of a sum of the model delta threshold value and an offset and the model delta value indicated by the model delta information.

3. The apparatus according to claim 1 or claim 2, comprising means for performing: receiving further information relating to the machine learning model.

4. The apparatus according to claim 3, wherein the further information relating to the machine learning model comprises at least one of the following: model delta information relating to an updated machine learning model, performance information relating to the updated machine learning model or the updated machine learning model.

5. The apparatus according to claim 3 or claim 4, comprising means for performing: determining to update the machine learning model based on the further information; and providing an indication at least to the entity on which the machine learning model is configured to update the machine learning model.

6. The apparatus according to any of claims 4 or 5, wherein the further information comprises the updated machine learning model and comparing means for performing: determining to provide the updated machine learning model to at least one further entity on which the machine learning model is configured.

7. The apparatus according to any of claims 1 to 6, comprising means for receiving performance information and model delta information relating to the machine learning model from a plurality of entities on which the machine learning model is configured and performing: determining a confidence value for a correspondence between the received model delta information and performance.

8. The apparatus according to any of claims 1 to 7, comprising means for: providing a configuration to the entity on which the machine learning model is configured to configure machine learning model delta parameters.

9. The apparatus according to any of claims 1 to 8 wherein the entity on which the machine learning model is configured comprises a user equipment or a network function.

10. An apparatus comprising means for: determining to update a machine learning model configured at the apparatus; updating the machine learning model; determining model delta information based on the machine learning model and the updated machine learning model; determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value; and providing the information relating to the machine learning model to the network entity.

11. The apparatus according to claim 10, comprising means for: further determining performance information based on the machine learning model and the updated machine learning model; providing the model delta information and the performance information to a network entity; and receiving an updated model delta threshold value from the network entity in response.

12. The apparatus according to claim 10 or claim 11 , wherein the information relating to the machine learning model comprises at least one of the following: the model delta information, performance information relating to the machine learning model or the updated machine learning model.

13. The apparatus according to any of claims 10 to 12, wherein the model delta information is determined at least based on a difference between weights of the machine learning model and weights of the updated machine learning model.

14. The apparatus according to claim 13, wherein the model delta information indicates at least one of a global machine learning delta value and a vector machine learning delta value.

15. A method comprising: receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured; determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model; and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

16. A method comprising: determining to update a machine learning model configured at the apparatus; updating the machine learning model; determining model delta information based on the machine learning model and the updated machine learning model; determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value; and providing the information relating to the machine learning model to the network entity.

17. An apparatus comprising at least one processor, and at least one memory storing instructions which, when executed by the processor, cause the apparatus at least to perform: receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured; determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model; and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

18. An apparatus comprising at least one processor, and at least one memory storing instructions which, when executed by the processor, cause the apparatus at least to perform: determining to update a machine learning model configured at the apparatus; updating the machine learning model; determining model delta information based on the machine learning model and the updated machine learning model; determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value; and providing the information relating to the machine learning model to the network entity.

19. A computer readable medium comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: receiving performance information and model delta information relating to a machine learning model from an entity on which the machine learning model is configured; determining to update a model delta threshold value based at least on the received performance information and model delta information and share an updated model delta threshold with the entity, wherein the model delta threshold is suitable for determining, by the entity on which the machine learning model is configured, whether to share further information relating to the machine learning model; and providing an indication of the updated model delta threshold value to the entity on which the machine learning model is configured.

20. A computer readable medium comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: determining to update a machine learning model configured at the apparatus; updating the machine learning model; determining model delta information based on the machine learning model and the updated machine learning model; determining to provide information relating to the machine learning model to a network entity based on a comparison between the model delta information and a model delta threshold value; and providing the information relating to the machine learning model to the network entity.