WO2025008083A1 - Performance monitoring for machine learning model - Google Patents

Abstract

Various aspects of the present disclosure relate to a network entity storing code comprising instructions executable to cause the network entity to: receive a message comprising an identifier of a machine learning (ML) model and an associated performance threshold; obtain the ML model based at least in part on the identifier of the ML model; determine whether the ML model's performance satisfies the performance threshold associated with the ML model; and transmit a message comprising an indication of whether the performance of the ML model satisfies the performance threshold. Also disclosed is a network entity storing code comprising instructions executable to cause the network entity to: transmit a message comprising an identifier of an ML model and an associated performance threshold; and receive a message comprising an indication of whether the ML model's performance satisfies the performance threshold. Associated methods for both network entities are also disclosed.

Description

PERFORMANCE MONITORING FOR MACHINE LEARNING MODEL

TECHNICAL FIELD

[0001] The present disclosure relates to wireless communications, and more specifically to the use of machine learning (ML) models in wireless communications.

BACKGROUND

[0002] A wireless communications system may include one or multiple network communication devices, such as base stations, which may support wireless communications for one or multiple user communication devices, which may be otherwise known as user equipment (UE), or other suitable terminology. The wireless communications system may support wireless communications with one or multiple user communication devices by utilizing resources of the wireless communication system (e.g., time resources (e.g., symbols, slots, subframes, frames, or the like) or frequency resources (e.g., subcarriers, carriers, or the like). Additionally, the wireless communications system may support wireless communications across various radio access technologies including third generation (3G) radio access technology, fourth generation (4G) radio access technology, fifth generation (5G) radio access technology, among other suitable radio access technologies beyond 5G (e.g., sixth generation (6G)).

SUMMARY

[0003] An article “a” before an element is unrestricted and understood to refer to “at least one” of those elements or “one or more” of those elements. The terms “a,” “at least one,” “one or more,” and “at least one of one or more” may be interchangeable. As used herein, including in the claims, “or” as used in a list of items (e.g., a list of items prefaced by a phrase such as “at least one of’ or “one or more of’ or “one or both of’) indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Also, as used herein, the phrase “based on” shall not be constmed as a reference to a closed set of conditions. For example, an example step that is described as “based on condition A” may be based on both a condition A and a condition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase “based on” shall be construed in the same manner as the phrase “based at least in part on. Further, as used herein, including in the claims, a “set” may include one or more elements.

[0004] Some implementations of the method and apparatuses described herein may include a network entity storing code comprising instructions executable to cause the network entity to: receive a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; obtain the ML model based at least in part on the identifier of the ML model; determine whether a performance of the ML model satisfies the performance threshold associated with the ML model; and transmit a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold.

[0005] The instructions may be further executable to cause the network entity to: obtain model data and comparison data; and compare the model data with the comparison data, wherein to determine whether the performance of the ML model satisfies the performance threshold associated with the ML model is based at least in part on the comparison of the model data with the comparison data.

[0006] The model data may comprise a current accuracy of a location determined using the ML model and the comparison data comprises a previous accuracy of a location previously determined by the ML model.

[0007] The model data may comprise a current accuracy of a location determined using the ML model and the comparison data comprises a previous accuracy of a location determined by other positioning methods.

[0008] The instructions may be further executable to cause the network entity to obtain the previous accuracy from one or more of an analytic data repository function (ADRF) entity or an operations, administration and maintenance (0AM) entity.

[0009] The instructions may be further executable to cause the network entity to obtain the accuracy of the location determined using the ML model from a second network entity. [0010] The model data may comprise a location determined using the ML model and the comparison data comprises a known value of the location.

[0011] The instructions may be further executable to cause the network entity to obtain the known value of the location from one or more of: a UE, a radio access network (RAN) node, or an operations, administration and maintenance (0AM) entity.

[0012] The model data may comprise a statistical parameter of training data of the ML model, and wherein the comparison data comprises a statistical parameter of corresponding current data.

[0013] The instructions may be further executable to cause the network entity to obtain the training data of the ML model from a database accessible to the network entity.

[0014] The instructions may be further executable to cause the network entity to obtain a replacement ML model based at least in part on the performance of the ML model failing to satisfy the performance threshold.

[0015] The instructions may be further executable to cause the network entity to obtain the replacement ML model by training the replacement ML model to satisfy the performance threshold.

[0016] The response message may comprise one or more of the replacement ML model or an indication of a network location where the replacement ML model can be accessed.

[0017] The instructions may be further executable to cause the network entity to determine whether the performance of the ML model satisfies the performance threshold associated with the ML model for at least a set of one or more time instances or during at least one duration, wherein the request message indicates the set of one or more time instances or the at least one duration.

[0018] Some implementations of the method and apparatuses described herein may further include a method performed by a network entity, the method comprising: receiving a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; obtaining the ML model based at least in part on the identifier of the ML model; determining whether a performance of the ML model satisfies the performance threshold associated with the ML model; and transmitting a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold.

[0019] Some implementations of the method and apparatuses described herein may further include a network entity storing code comprising instructions executable to cause the network entity to: transmit a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; and receive a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold.

[0020] The network entity may use the ML model to determine positions, and the instructions may be further executable to cause the network entity to: receive a request message for comparison data; obtain the requested comparison data; and transmit a response message comprising the requested comparison data.

[0021] The instructions may be further executable to cause the network entity to obtain the requested comparison data by, each time the network entity uses the ML model to determine a position, recording the determined position and/or an accuracy of the determined position.

[0022] The instructions may be further executable to cause the network entity to obtain the requested comparison data by, each time the network entity determines a position, recording the determined position and/or an accuracy of the determined position.

[0023] Some implementations of the method and apparatuses described herein may further include a method performed by a network entity, the method comprising: transmitting a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; and receiving a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold.

[0024] Some implementations of the method and apparatuses described herein may further include a network entity configured to: receive a request message for comparison data for evaluating the performance of an ML model used by the network entity; obtain the requested comparison data; and transmit a response message comprising the requested comparison data.

[0025] The network entity may be configured to obtain the requested comparison data by, each time the network entity uses the ML model to determine a position, recording the determined position and/or an accuracy of the determined position.

[0026] The network entity may be configured to obtain the requested comparison data by, each time the network entity determines a position, recording the determined position and/or an accuracy of the determined position.

[0027] Some implementations of the method and apparatus described herein may further include a method performed by a network entity, the method comprising: receiving a request message for comparison data for evaluating the performance of an ML model used by the network entity; obtaining the requested comparison data; and transmitting a response message comprising the requested comparison data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 illustrates an example of a wireless communications system in accordance with aspects of the present disclosure.

[0029] Figure 2 illustrates an example of direct AIML positioning.

[0030] Figures 3A-3C illustrate an example of assisted AIML positioning.

[0031] Figure 4 illustrates a network architecture.

[0032] Figure 5 illustrates a network architecture in accordance with aspects of the present disclosure.

[0033] Figure 6 illustrates a process for monitoring the performance of an ML model in accordance with aspects of the present disclosure.

[0034] Figure 7 illustrates an example of a UE in accordance with aspects of the present disclosure.

[0035] Figure 8 illustrates an example of a processor in accordance with aspects of the present disclosure. [0036] Figure 9 illustrates an example of a network equipment (NE) in accordance with aspects of the present disclosure.

[0037] Figure 10 illustrates a method for monitoring the performance of an ML model in accordance with aspects of the present disclosure.

[0038] Figure 11 illustrates a further method for monitoring the performance of an ML model in accordance with aspects of the present disclosure.

DETAILED DESCRIPTION

[0039] A wireless communications system, including one or more communication devices may be enabled (e.g. configured) to support ML, and more generally artificial intelligence (Al) (referred to collectively as AIML) for various applications or services associated with the wireless communications system. For example, one or more communication devices (e.g. a base station or other network entity or UE) may be configured to support AIML for improving positioning, such as locating a UE in the wireless communications system.

[0040] Some wireless communication system may support positioning (e.g. positioning operations, positioning tasks, positioning procedures) by a network function, such as a location management function (LMF). For example, the LMF may be queried (e.g. requested) by a communication device (e.g. a base station or other network entity) to provide for a location of a communication device, such as a UE. The LMF may be configured to perform one or more positioning operations, positioning tasks, or positioning procedures to determine and report the location of the communication device.

[0041] Some wireless communication systems may support AIML-assisted positioning, which may involve adjusting (e.g. modifying, refining, updating) the reported location of the communication device by the LMF using an ML model implemented (e.g. located, executed, stored) remote in these wireless communication systems.

[0042] Although the ML-based positioning of these wireless communication systems may provide some improvements to positioning and reporting, the ML-based positioning of these wireless communication systems may experience a degradation of performance of the ML models and, as a result, impact the reliability (e.g. accuracy) of the positioning and reporting. For example, network conditions may change over time, rendering an original training of an ML model invalid.

[0043] The implementation of schemes for monitoring AIML models in use for telecommunications may allow for improved accuracy and consistency of model outputs. For example, in the context of positioning, it may become possible to provide more accurate locations for a longer period of time.

[0044] Additionally, monitoring schemes as disclosed herein may allow for increased flexibility of a communications system (e.g. in regard to positioning) when network or other conditions change, allowing for the updating or replacement of ML models that are found to no longer operate adequately in the new conditions.

[0045] Additionally, monitoring the performance of ML models can allow for greater consistency of results. For example, in the context of positioning, model monitoring may allow for more consistent location accuracies across different environments.

[0046] The term “direct AIML positioning” may be used to refer to cases where a UE location is the output of an AIML model, with the assumption that there is an ML model at the UE or the LMF that is used to predict a location. “AIML assisted positioning” may refer to cases wherein an AI/ML model output can be a new measurement and/or an enhancement of an existing measurement, wherein e.g. AI/ML models are used by a UE or a gNB.

[0047] Within this framework, AIML positioning may be categorized as follows.

[0048] Case 1 : UE-based positioning with UE-side model, direct AI/ML or AI/ML assisted positioning.

[0049] Case 2a: UE-assisted/LMF -based positioning with UE-side model, AI/ML assisted positioning.

[0050] Case 2b: UE-assisted/LMF -based positioning with LMF-side model, direct

AI/ML positioning. [0051] Case 3a: NG-RAN node assisted positioning with gNB-side model, AI/ML assisted positioning.

[0052] Case 3b: NG-RAN node assisted positioning with LMF-side model, direct AI/ML positioning.

[0053] In some examples, “one-sided” model use is prioritised, wherein inference (the use of AIML models) is performed entirely at a UE or at the network.

[0054] In particular, case 1 is mainly focused on using UE-side models for Direct AI/ML or AI/ML assisted positioning.

[0055] Cases 2a and 3 a indicate that AI/ML assisted positioning is mainly used with UE-side and/or gNB-side ML models, primarily to enhance the measurements provided at the LMF for deriving location estimates (e.g. more accurate measurements taking into account NLOS conditions). Since the ML model needs to be trained either at UE(-side) or gNB(-side), this requires defining data collection mechanisms for UE and RAN as well as procedure for model delivery/transfer to UE/gNB and model identification/management.

[0056] Cases 2b and 3 b indicate that LMF-side models are used for Direct AI/ML positioning either by collecting raw data or "AI/ML enhanced data" from UE and/or gNB to derive a location. The ML-model is trained at LMF-side (or CN in general). The LMF-side models used can be trained either with raw data or AI/ML enhanced data.

[0057] Figure 2 illustrates an example of direct AIML positioning.

[0058] Figures 3A-3C illustrate various implementations of assisted AIML positioning.

[0059] Figure 4 illustrates a network architecture that may allow data collection relating to the use of network data analytics functions (NWDAFs). In some cases, an AnLF requests a trained ML model from the MTLF by including in the request an Analytic ID corresponding to the Analytic ID requested by an analytics consumer. Information on the available Analytic ID(s) supported by an AnLF is provided in 3GPP TS 23.288. The MTLF trains an ML model corresponding to the Analytic ID requested by collecting data from one or more data sources (NF, 0AM or UEs). [0060] Aspects of the present disclosure are described in the context of a wireless communications system.

[0061] Figure 1 illustrates an example of a wireless communications system 100 in accordance with aspects of the present disclosure. The wireless communications system 100 may include one or more NE 102, one or more UE 104, and a core network (CN) 106. The wireless communications system 100 may support various radio access technologies. In some implementations, the wireless communications system 100 may be a 4G network, such as an LTE network or an LTE- Advanced (LTE-A) network. In some other implementations, the wireless communications system 100 may be a NR network, such as a 5G network, a 5G- Advanced (5G-A) network, or a 5G ultrawideband (5G-UWB) network. In other implementations, the wireless communications system 100 may be a combination of a 4G network and a 5G network, or other suitable radio access technology including Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20. The wireless communications system 100 may support radio access technologies beyond 5G, for example, 6G. Additionally, the wireless communications system 100 may support technologies, such as time division multiple access (TDMA), frequency division multiple access (FDMA), or code division multiple access (CDMA), etc.

[0062] The one or more NE 102 may be dispersed throughout a geographic region to form the wireless communications system 100. One or more of the NE 102 described herein may be or include or may be referred to as a network node, a base station, a network element, a network function, a network entity, a radio access network (RAN), a NodeB, an eNodeB (eNB), a next-generation NodeB (gNB), or other suitable terminology. An NE 102 and a UE 104 may communicate via a communication link, which may be a wireless or wired connection. For example, an NE 102 and a UE 104 may perform wireless communication (e.g., receive signaling, transmit signaling) over a Uu interface.

[0063] An NE 102 may provide a geographic coverage area for which the NE 102 may support services for one or more UEs 104 within the geographic coverage area. For example, an NE 102 and a UE 104 may support wireless communication of signals related to services (e.g., voice, video, packet data, messaging, broadcast, etc.) according to one or multiple radio access technologies. In some implementations, an NE 102 may be moveable, for example, a satellite associated with a non-terrestrial network (NTN). In some implementations, different geographic coverage areas 112 associated with the same or different radio access technologies may overlap, but the different geographic coverage areas may be associated with different NE 102.

[0064] The one or more UE 104 may be dispersed throughout a geographic region of the wireless communications system 100. A UE 104 may include or may be referred to as a remote unit, a mobile device, a wireless device, a remote device, a subscriber device, a transmitter device, a receiver device, or some other suitable terminology. In some implementations, the UE 104 may be referred to as a unit, a station, a terminal, or a client, among other examples. Additionally, or alternatively, the UE 104 may be referred to as an Internet-of- Things (loT) device, an Intemet-of-Everything (loE) device, or machine-type communication (MTC) device, among other examples.

[0065] A UE 104 may be able to support wireless communication directly with other UEs 104 over a communication link. For example, a UE 104 may support wireless communication directly with another UE 104 over a device-to-device (D2D) communication link. In some implementations, such as vehicle-to-vehicle (V2V) deployments, vehicle-to-everything (V2X) deployments, or cellular-V2X deployments, the communication link 114 may be referred to as a sidelink. For example, a UE 104 may support wireless communication directly with another UE 104 over a PC5 interface.

[0066] An NE 102 may support communications with the CN 106, or with another NE

102, or both. For example, an NE 102 may interface with other NE 102 or the CN 106 through one or more backhaul links (e.g., SI, N2, N2, or network interface). In some implementations, the NE 102 may communicate with each other directly. In some other implementations, the NE 102 may communicate with each other or indirectly (e.g., via the CN 106. In some implementations, one or more NE 102 may include subcomponents, such as an access network entity, which may be an example of an access node controller (ANC). An ANC may communicate with the one or more UEs 104 through one or more other access network transmission entities, which may be referred to as a radio heads, smart radio heads, or transmission-reception points (TRPs). [0067] The CN 106 may support user authentication, access authorization, tracking, connectivity, and other access, routing, or mobility functions. The CN 106 may be an evolved packet core (EPC), or a 5G core (5GC), which may include a control plane entity that manages access and mobility (e.g., a mobility management entity (MME), an access and mobility management functions (AMF)) and a user plane entity that routes packets or interconnects to external networks (e.g., a serving gateway (S-GW), a Packet Data Network (PDN) gateway (P-GW), or a user plane function (UPF)). In some implementations, the control plane entity may manage non-access stratum (NAS) functions, such as mobility, authentication, and bearer management (e.g., data bearers, signal bearers, etc.) for the one or more UEs 104 served by the one or more NE 102 associated with the CN 106.

[0068] The CN 106 may communicate with a packet data network over one or more backhaul links (e.g., via an SI, N2, N2, or another network interface). The packet data network may include an application server. In some implementations, one or more UEs 104 may communicate with the application server. A UE 104 may establish a session (e.g., a protocol data unit (PDU) session, or the like) with the CN 106 via an NE 102. The CN 106 may route traffic (e.g., control information, data, and the like) between the UE 104 and the application server using the established session (e.g., the established PDU session). The PDU session may be an example of a logical connection between the UE 104 and the CN 106 (e.g., one or more network functions of the CN 106).

[0069] In the wireless communications system 100, the NEs 102 and the UEs 104 may use resources of the wireless communications system 100 (e.g., time resources (e.g., symbols, slots, subframes, frames, or the like) or frequency resources (e.g., subcarriers, carriers)) to perform various operations (e.g., wireless communications). In some implementations, the NEs 102 and the UEs 104 may support different resource structures. For example, the NEs 102 and the UEs 104 may support different frame structures. In some implementations, such as in 4G, the NEs 102 and the UEs 104 may support a single frame structure. In some other implementations, such as in 5 G and among other suitable radio access technologies, the NEs 102 and the UEs 104 may support various frame structures (i.e., multiple frame structures). The NEs 102 and the UEs 104 may support various frame structures based on one or more numerologies. [0070] One or more numerologies may be supported in the wireless communications system 100, and a numerology may include a subcarrier spacing and a cyclic prefix. A first numerology (e.g., /r=0) may be associated with a first subcarrier spacing (e.g., 15 kHz) and a normal cyclic prefix. In some implementations, the first numerology (e.g., /r=0) associated with the first subcarrier spacing (e.g., 15 kHz) may utilize one slot per subframe. A second numerology (e.g., /r=l) may be associated with a second subcarrier spacing (e.g., 30 kHz) and a normal cyclic prefix. A third numerology (e.g., /r=2) may be associated with a third subcarrier spacing (e.g., 60 kHz) and a normal cyclic prefix or an extended cyclic prefix. A fourth numerology (e.g., /r=3) may be associated with a fourth subcarrier spacing (e.g., 120 kHz) and a normal cyclic prefix. A fifth numerology (e.g., /r=4) may be associated with a fifth subcarrier spacing (e.g., 240 kHz) and a normal cyclic prefix.

[0071] A time interval of a resource (e.g., a communication resource) may be organized according to frames (also referred to as radio frames). Each frame may have a duration, for example, a 10 millisecond (ms) duration. In some implementations, each frame may include multiple subframes. For example, each frame may include 10 subframes, and each subframe may have a duration, for example, a 1 ms duration. In some implementations, each frame may have the same duration. In some implementations, each subframe of a frame may have the same duration.

[0072] Additionally or alternatively, a time interval of a resource (e.g., a communication resource) may be organized according to slots. For example, a subframe may include a number (e.g., quantity) of slots. The number of slots in each subframe may also depend on the one or more numerologies supported in the wireless communications system 100. For instance, the first, second, third, fourth, and fifth numerologies (i.e., /r=0, jU=l , /r=2, jU=3, /r=4) associated with respective subcarrier spacings of 15 kHz, 30 kHz, 60 kHz, 120 kHz, and 240 kHz may utilize a single slot per subframe, two slots per subframe, four slots per subframe, eight slots per subframe, and 16 slots per subframe, respectively. Each slot may include a number (e.g., quantity) of symbols (e.g., OFDM symbols). In some implementations, the number (e.g., quantity) of slots for a subframe may depend on a numerology. For a normal cyclic prefix, a slot may include 14 symbols. For an extended cyclic prefix (e.g., applicable for 60 kHz subcarrier spacing), a slot may include 12 symbols. The relationship between the number of symbols per slot, the number of slots per subframe, and the number of slots per frame for a normal cyclic prefix and an extended cyclic prefix may depend on a numerology. It should be understood that reference to a first numerology (e.g., /r=0) associated with a first subcarrier spacing (e.g., 15 kHz) may be used interchangeably between subframes and slots.

[0073] In the wireless communications system 100, an electromagnetic (EM) spectrum may be split, based on frequency or wavelength, into various classes, frequency bands, frequency channels, etc. By way of example, the wireless communications system 100 may support one or multiple operating frequency bands, such as frequency range designations FR1 (410 MHz - 7.125 GHz), FR2 (24.25 GHz - 52.6 GHz), FR3 (7.125 GHz - 24.25 GHz), FR4 (52.6 GHz - 114.25 GHz), FR4a or FR4-1 (52.6 GHz - 71 GHz), and FR5 (114.25 GHz - 300 GHz). In some implementations, the NEs 102 and the UEs 104 may perform wireless communications over one or more of the operating frequency bands. In some implementations, FR1 may be used by the NEs 102 and the UEs 104, among other equipment or devices for cellular communications traffic (e.g., control information, data). In some implementations, FR2 may be used by the NEs 102 and the UEs 104, among other equipment or devices for short-range, high data rate capabilities.

[0074] FR1 may be associated with one or multiple numerologies (e.g., at least three numerologies). For example, FR1 may be associated with a first numerology (e.g., /r=0), which includes 15 kHz subcarrier spacing; a second numerology (e.g., /r=l), which includes 30 kHz subcarrier spacing; and a third numerology (e.g., /r=2), which includes 60 kHz subcarrier spacing. FR2 may be associated with one or multiple numerologies (e.g., at least 2 numerologies). For example, FR2 may be associated with a third numerology (e.g., /r=2), which includes 60 kHz subcarrier spacing; and a fourth numerology (e.g., /r=3), which includes 120 kHz subcarrier spacing.

[0075] Figure 5 shows an example of a network architecture 500 in accordance with aspects of the present disclosure. In some implementations, the network architecture 500 may implement or be implemented by aspects of the wireless communication system 100 described herein with reference to Figure 1. [0076] The network architecture 500 comprises a first LMF 502 and a second LMF 504, also referred to herein as an LMF-T (training LMF) 502 and an LMF-I (inference LMF) 504. The two LMFs 502, 504 may able to contact a network exposure function (NEF) 516. Each LMF 502, 504 may also be able to access various data sources 512, 514, such as e.g. an analytic data repository function (ADRF) and various administration and maintenance (0 AM) processes of the communications network. The network architecture 500 may further comprise one or more user equipment (UEs) devices 506 (which may correspond to a UE 104 as described herein with reference to Figure 1), as well as one or more radio access network (RAN) nodes 508 (which may correspond to a NE 102 as described herein with reference to Figure 1). Each LMF 502, 504 may be able to access one or more UEs 506 and one or more RAN nodes 508. The network architecture 500 may also comprise an access and mobility management function (AMF) 510, which may be accessible to each of the LMFs 502, 504.

[0077] The LMF-T 502 may be configured to train ML models. The LMF-T 502 may also have access to one or more existing ML models which have been trained previously. These existing models may, for example, be stored in various locations that are accessible via the network.

[0078] The LMF-T 502 may be configured to obtain training data for training ML models. For example, the LMF-T 202 may access data sources 514 as described above, as well as e.g. UEs 506 and RAN nodes 508 of the network, to obtain training data. This training data may generally be positioning data, such as for example reference signal information (as generally defined in, for example, 3GPP TS 37.355 and 3GPP TS 38.355).

[0079] For example, the LMF-T 502 may be configured to train ML models suitable for use in direct AIML positioning and/or assisted AIML positioning.

[0080] The LMF-T 502 may be configured to accept requests for monitoring the performance of ML models that were trained by the LMF-T 502. This functionality is described in more detail below with reference to Figure 6.

[0081] The LMF-I 504 may be configured to use ML models for positioning, e.g. for determining the location of a UE 506. For example, the LMF-I may be configured to use direct AIML positioning and/or assisted AIML positioning. The LMF-I may also be capable of using non-AIML-based positioning methods.

[0082] Merely for context, it is noted that positioning methods supported by the LMF-I 504 may include any or all of: A-GNSS, OTDOA, E-CID, Sensor, WLAN, Bluetooth, TBS, DL-TDOA, DL-AoD, Multi-RTT, NR E-CID, UL-TDOA, and/or UL-AoA.

[0083] The LMF-I 504 may be provisioned with one or more ML models for use in positioning. Each model may be identified by a model ID that serves as a unique identifier for that model. In an example, each model used by the LMF-I 504 may be trained to meet different respective requirements, in order to allow for direct and/or assisted AIML positioning in different situations. Such requirements may include any or all of: the type of positioning method supported (e.g. TDOA, RTT, and/or sidelink positioning methods); support for direct AIML positioning with specific 2D/3D horizontal and/or vertical accuracy requirements; support for deriving a location within a specific response time; a requirement or lack of requirement for assistance signalling from PRUs and/or RAN nodes; support for determining positions using NLOS measurements reported from UEs, RAN nodes, and/or PRUs; support for determining positions using AIML assisted measurements reported from UEs, RAN node, and/or PRUs; support for different environment types, e.g. outdoor, indoor office, indoor factory; and/or support for different areas of a 3GPP network, e.g. tracking area, RAN area, PCI, NCGI, and/or Zone IDs.

[0084] The LMF-I 504 may be configured to contact the LMF-T 502 from which a particular ML model was obtained in order to request that the LMF-T 502 monitors the performance of the ML model and determines whether the performance meets a certain threshold specified by the LMF-I 504.

[0085] It is noted that, while the architecture 500 illustratively shows only a single LMF-I 504, it is envisioned that networks may in general comprise more than one LMF-I. In this case,

[0086] Figure 6 shows a process 600 for monitoring the performance of an ML model in accordance with aspects of the present disclosure. The process 600 may, for example, be implemented within the network architecture 500 of Figure 5. [0087] At step S602, an LMF-I 504 of the communications system 100 may send a request message (otherwise referred to herein as a monitoring request) to an LMF-T 502 requesting the LMF-T 502 to monitor the performance of an ML model in use by the LMF- I 504. It is generally envisioned that the LMF-I 504 will send this request to the same LMF- T 502 that originally trained the ML model in question.

[0088] The request message may contain a model ID associated with the model. The model ID may generally be a unique identifier of the particular model, such as a version number.

[0089] The request message may further include an indication of a performance threshold that the ML model is required to meet. For example, in the context of positioning, the request message may specify a particular positioning accuracy that is required of the ML model.

[0090] Merely for context, an exemplary definition of position accuracy is briefly given. When a location is requested from the LMF-I 504, the location request may generally comprise a required precision (for example, with Im or within 1km). The LMF-I 504 may then determine the requested location using a positioning method (for example, direct AIML positioning using an ML model). The LMF-I 504 may then respond to the location request giving the determined location together with an estimated precision. The accuracy of the positioning method (for example, of the ML model used for direct AIML positioning) may then be defined as the percentage of requests for which the ML model achieves the required precision. For example, if a particular client always requests locations within 10m, and the ML model is able to return a position within a precision of 10m 85% of the time, the ML model may be said to be 85% accurate.

[0091] Additionally or alternatively, the performance threshold may specify a maximum acceptable difference between historical and current position determinations made using the ML model.

[0092] Additionally or alternatively, the performance threshold may specify a maximum acceptable difference between historical data that was originally used to train the ML model, and an updated real-time version of the same dataset. For example, the performance threshold may specify how much a particular statistical parameter of the training data may have changed.

[0093] The request message may further indicate when the ML model should be evaluated (at a particular time or times, and/or during one or more time periods).

[0094] The request message may further indicate that the ML model should be evaluated for location requests within a specific geographic region. For example, the request may specify that the accuracy of positions determined by the ML model must meet a particular threshold for positions within the specific region.

[0095] In some examples, the LMF-I 502 may request the LMF-T 504 to provide a new ML model, and the request message may be a request for monitoring the performance of the newly -provided ML model. In these examples, the ML model’s performance may be monitored from the start of its use by the LMF-I 502.

[0096] At step S604, the LMF-T 502 may obtain the ML model based at least in part on the identifier of the ML model referred to in the request, for example using the model ID included in the request. For example, the LMF-T 502 may contact an analytics data repository function (ADRF) of the core network 106 and request a copy of the ML model by supplying the model ID.

[0097] Alternatively, in some cases the LMF-T 502 may already have the ML model stored in a location accessible to the LMF-T 502. In such cases the LMF-T 502 may simply access the model without necessarily contacting an ADRF.

[0098] The LMF-T 502 may then identify the capabilities of the ML model in order to determine how the model’s performance should be monitored. For example, the LMF-T 502 may ascertain any or all of: which positioning method or methods the ML model is configured to use; how precise the ML model is capable of being; what radio environment conditions the ML model is configured to be used for; and/or what training data was used to train the ML model originally. Regarding training data, it is noted that if, as generally envisioned, the LMF-T 502 originally trained the ML model in the first place, the LMF-T may then have a record of the training data that was originally used. For example, this record may be stored in memory accessible to the LMF-T 502. [0099] If the LMF-T 502 discovers that the model identified in the request does not exist - for example, if the LMF-T 502 supplies the model ID from the request to an ADRF and the ADRF responds that there is no known model with that ID - the LMF-T 502 may optionally respond to the LMF-I 504 indicating that the model does not exist.

[0100] At step S606, the LMF-T 502 may optionally contact various data sources to obtain information for use in evaluating the ML model’s performance. For example, the LMF-T 502 may contact any or all of an analytics data repository function 514 of the core network 106; an operations, administration and maintenance (0 AM) entity 514 of the core network 106; an analytics management function 510 of the core network 106; a UE 506 of the communications system 100; a radio access network (RAN) node 508 of the communications system 100; and/or a gateway mobile location centre (GMLC) of the communications system 100.

[0101] For example, the LMF-T 502 may contact the 0AM and/or ADRF 514 to retrieve historical location information. In particular, the LMF-T 502 may retrieve historical locations of UEs 506 that are located within an area served by the LMF-I 504 that made the request. Where the historical locations were found using ML models, including the ML model that is the subject of the request, the LMF-T 502 may retrieve the model ID of the model used to determine the historical location, together with information on the positioning method or methods used.

[0102] Additionally or alternatively, the LMF-T 502 may contact one or more UEs 506, RAN nodes 508, and/or 0AM 514 to obtain real-time positioning measurement data. For example, the LMF-T 502 may retrieve real-time positions of UEs 506 that are located within an area served by the LMF-I 504 that made the request.

[0103] The data retrieved by the LMF-T 502 from any of the sources so far identified may include channel fingerprint information.

[0104] Additionally or alternatively, the LMF-T 502 may request information on UEs 506 with known locations (which may be referred to as positioning reference units, PRUs).

[0105] At step S608, the LMF-T 502 may optionally contact the LMF-I 504 to request positioning information. [0106] For simplicity, Figure 6 illustrates a case wherein the LMF-T 502 contacts the same LMF-I 504 that requested monitoring of the ML model. However, since (as noted above) the communications system 100 may comprise multiple LMF-Is, it is generally envisioned that the LMF-T 502 may contact any or all LMF-Is that use the same ML model referred to in the original request and serve the same area as the LMF-I 504. This may include the LMF-I 504 that made the request, but may also include other LMF-Is that did not make any request. In particular, at step S608 the LMF-T 502 may contact an LMF-I that is not the LMF-I 504 that sent the monitoring request, but rather a different LMF-I using the same ML model.

[0107] The LMF-T 502 may request that the LMF-I 504 sends information to the LMF- T 502 when the LMF-I 504 determines a position during normal operation.

[0108] In one example, the LMF-T 502 may request that the LMF-I 504 sends information to the LMF-T 502 whenever the LMF-I 504 determines a position using direct AIML positioning based on the ML model that is the subject of the monitoring request. This information may include the determined position, as well as a precision associated with the position and an associated position accuracy calculated by the LMF-I 504. In this case, the LMF-T 502 may include the model ID in its request.

[0109] In another example, the LMF-T 502 may request that the LMF-I 504 sends information to the LMF-T 502 whenever the LMF-I 504 determines a position using any method, whether or not the ML model is used. This may give the LMF-T 502 information on positions determined using the ML model, and on positions determined using other methods not involving the ML model.

[0110] Additionally or alternatively, the LMF-T 502 may contact a gateway mobile location centre to request the above-identified information.

[0111] At step S610, the LMF-I 504 may determine locations as part of the LMF-I 504’ s normal function (for example, in response to location requests from clients). The LMF-I 504 may use any positioning method to make such determinations. This may include using the ML model for direct AIML positioning and/or AIML-assisted positioning. The LMF-I 504 may also use legacy (non-AIML) methods to determine positions that do not involve the ML model.

[0112] At step S612, the LMF-I 504 may respond to the request sent by the LMF-T 502 at step S608 by providing the requested information. This response may take the form of a single message including information obtained over several positioning operations. Alternatively, the LMF-I 504 may send a separate message LMF-T 502 each time the LMF-I 504 handles a positioning request using a method consistent with the request of the LMF-T 502.

[0113] At step S614, the LMF-T 502 may evaluate whether the performance of the ML model meets the performance threshold that was included in the original request sent at step S602. This may be done in one or more of the following ways.

[0114] As a first example, the LMF-T 502 may evaluate the performance of the ML model by comparing the accuracy of positions determined by the ML model using direct AIML positioning (for example, positions reported by the LMF-I 504) against the accuracy of historical positions determined using the same ML model (for example, retrieved from the 0AM and/or ADRF 514). In this case, the performance threshold as described above may be a maximum acceptable change in the accuracy of locations determined using the model.

[0115] Additionally or alternatively, the LMF-T 502 may evaluate the performance of the ML model by comparing the accuracy of positions determined by the ML model using direct AIML positioning (for example, positions reported by the LMF-I 504) against the accuracy of positions obtained using alternative methods that do not use the ML model (for example, positions reported by the LMF-I 504 and/or real-time positions reported by UEs 506, RAN nodes 508, and/or 0AM 514). In this case, the performance threshold as described above may be a maximum acceptable discrepancy between the accuracy obtained by the ML model and the accuracy of positions obtained using alternative methods that do not use the ML model. Additionally or alternatively, the performance threshold may indicate that the ML model is required to provide superior results to alternative non-AIML methods. [0116] Additionally or alternatively, the LMF-T 502 may evaluate the performance of the ML model by comparing positions determined by the ML model using direct AIML positioning (for example, positions reported by the LMF-I 504) for UEs that have known locations (e.g. PRUs), against the known locations of those UEs to calculate an accuracy. In this case, the performance threshold as described above may be a maximum acceptable discrepancy between the positions determined by the ML model and the known positions.

[0117] Additionally or alternatively, the LMF-T 502 may evaluate the performance of the ML model by comparing training data that was used to train the ML model against a current version of the same dataset. For example, the LMF-T 502 may determine a statistical parameter (such as e.g. mean and/or standard deviation) of the training data, and compare this statistical parameter to the same statistical parameter calculated for an up-to- date version of the training data. As an example, the training data may have been channel fingerprint information gathered at the time when the ML model was trained. In this case, the LMF-T 502 may compare a statistical parameter of this historical channel fingerprint information against up-to-date channel fingerprint information. The LMF-T 502 may then determine that the ML model is no longer suitable and must be retrained and/or replaced if the statistical parameter has changed significantly since the model was trained. In this case, the performance threshold as described above may be a maximum acceptable change of the statistical parameter.

[0118] If the LMF-T 502 determines using any or all of the above methods that the ML model’s performance does not meet the performance threshold, the LMF-T 502 may optionally train a replacement ML model that does meet the performance threshold.

[0119] In any or all of these examples, the LMF-T 502 may additionally compare the accuracies of positions determined by the ML model (for example, as reported by the LMF- I 504 and/or a GMLC as described above) directly against the performance threshold, in the case where the performance threshold specifies a minimum acceptable positioning accuracy as described above.

[0120] At step S616, if it has been determined that the ML model does not meet the performance threshold, the LMF-T 502 may notify the LMF-I 504 of this outcome. It is generally envisioned that this notification may not be sent if the LMF-T 502 determines that the ML model’s performance does meet the performance threshold.

[0121] Additionally, if the LMF-T 502 trained a replacement model as described above, the LMF-T 502 may provide the replacement model to the LMF-I 504. This may be done by transmitting the ML model file to the LMF-I 504, and/or by providing an indication of a network location where the ML model is accessible to the LMF-I 504.

[0122] Figure 7 illustrates an example of a UE 700 in accordance with aspects of the present disclosure. The UE 700 may include a processor 702, a memory 704, a controller 706, and a transceiver 708. The processor 702, the memory 704, the controller 706, or the transceiver 708, or various combinations thereof or various components thereof may be examples of means for performing various aspects of the present disclosure as described herein. These components may be coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces.

[0123] The processor 702, the memory 704, the controller 706, or the transceiver 708, or various combinations or components thereof may be implemented in hardware (e.g., circuitry). The hardware may include a processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or other programmable logic device, or any combination thereof configured as or otherwise supporting a means for performing the functions described in the present disclosure.

[0124] The processor 702 may include an intelligent hardware device (e.g., a general- purpose processor, a DSP, a CPU, an ASIC, an FPGA, or any combination thereof). In some implementations, the processor 702 may be configured to operate the memory 704. In some other implementations, the memory 704 may be integrated into the processor 702.

The processor 702 may be configured to execute computer-readable instructions stored in the memory 704 to cause the UE 700 to perform various functions of the present disclosure.

[0125] The memory 704 may include volatile or non-volatile memory. The memory 704 may store computer-readable, computer-executable code including instructions when executed by the processor 702 cause the UE 700 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such the memory 704 or another type of memory. Computer-readable media includes both non- transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium may be any available medium that may be accessed by a general-purpose or special-purpose computer.

[0126] In some implementations, the processor 702 and the memory 704 coupled with the processor 702 may be configured to cause the UE 700 to perform one or more of the functions described herein (e.g., executing, by the processor 702, instructions stored in the memory 704). For example, the processor 702 may support wireless communication at the UE 700 in accordance with examples as disclosed herein. The UE 700 may be configured to support a means for one or more positioning methods, potentially including AIML-based positioning methods.

[0127] The controller 706 may manage input and output signals for the UE 700. The controller 706 may also manage peripherals not integrated into the UE 700. In some implementations, the controller 706 may utilize an operating system such as iOS®, ANDROID®, WINDOWS®, or other operating systems. In some implementations, the controller 706 may be implemented as part of the processor 702.

[0128] In some implementations, the UE 700 may include at least one transceiver 708. In some other implementations, the UE 700 may have more than one transceiver 708. The transceiver 708 may represent a wireless transceiver. The transceiver 708 may include one or more receiver chains 710, one or more transmitter chains 712, or a combination thereof.

[0129] A receiver chain 710 may be configured to receive signals (e.g., control information, data, packets) over a wireless medium. For example, the receiver chain 710 may include one or more antennas for receive the signal over the air or wireless medium. The receiver chain 710 may include at least one amplifier (e.g., a low-noise amplifier (LNA)) configured to amplify the received signal. The receiver chain 710 may include at least one demodulator configured to demodulate the receive signal and obtain the transmitted data by reversing the modulation technique applied during transmission of the signal. The receiver chain 710 may include at least one decoder for decoding the processing the demodulated signal to receive the transmitted data. [0130] A transmitter chain 712 may be configured to generate and transmit signals (e.g., control information, data, packets). The transmitter chain 712 may include at least one modulator for modulating data onto a carrier signal, preparing the signal for transmission over a wireless medium. The at least one modulator may be configured to support one or more techniques such as amplitude modulation (AM), frequency modulation (FM), or digital modulation schemes like phase-shift keying (PSK) or quadrature amplitude modulation (QAM). The transmitter chain 712 may also include at least one power amplifier configured to amplify the modulated signal to an appropriate power level suitable for transmission over the wireless medium. The transmitter chain 712 may also include one or more antennas for transmitting the amplified signal into the air or wireless medium.

[0131] Figure 8 illustrates an example of a processor 800 in accordance with aspects of the present disclosure. The processor 800 may be an example of a processor configured to perform various operations in accordance with examples as described herein. The processor 800 may include a controller 802 configured to perform various operations in accordance with examples as described herein. The processor 800 may optionally include at least one memory 804, which may be, for example, an L1/L2/L3 cache. Additionally, or alternatively, the processor 800 may optionally include one or more arithmetic-logic units (ALUs) 806. One or more of these components may be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces (e.g., buses).

[0132] The processor 800 may be a processor chipset and include a protocol stack (e.g., a software stack) executed by the processor chipset to perform various operations (e.g., receiving, obtaining, retrieving, transmitting, outputting, forwarding, storing, determining, identifying, accessing, writing, reading) in accordance with examples as described herein. The processor chipset may include one or more cores, one or more caches (e.g., memory local to or included in the processor chipset (e.g., the processor 800) or other memory (e.g., random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), static RAM (SRAM), ferroelectric RAM (FeRAM), magnetic RAM (MRAM), resistive RAM (RRAM), flash memory, phase change memory (PCM), and others). [0133] The controller 802 may be configured to manage and coordinate various operations (e.g., signaling, receiving, obtaining, retrieving, transmitting, outputting, forwarding, storing, determining, identifying, accessing, writing, reading) of the processor 800 to cause the processor 800 to support various operations in accordance with examples as described herein. For example, the controller 802 may operate as a control unit of the processor 800, generating control signals that manage the operation of various components of the processor 800. These control signals include enabling or disabling functional units, selecting data paths, initiating memory access, and coordinating timing of operations.

[0134] The controller 802 may be configured to fetch (e.g., obtain, retrieve, receive) instructions from the memory 804 and determine subsequent instruction(s) to be executed to cause the processor 800 to support various operations in accordance with examples as described herein. The controller 802 may be configured to track memory address of instructions associated with the memory 804. The controller 802 may be configured to decode instructions to determine the operation to be performed and the operands involved. For example, the controller 802 may be configured to interpret the instruction and determine control signals to be output to other components of the processor 800 to cause the processor 800 to support various operations in accordance with examples as described herein. Additionally, or alternatively, the controller 802 may be configured to manage flow of data within the processor 800. The controller 802 may be configured to control transfer of data between registers, arithmetic logic units (ALUs), and other functional units of the processor 800.

[0135] The memory 804 may include one or more caches (e.g., memory local to or included in the processor 800 or other memory, such RAM, ROM, DRAM, SDRAM, SRAM, MRAM, flash memory, etc. In some implementations, the memory 804 may reside within or on a processor chipset (e.g., local to the processor 800). In some other implementations, the memory 804 may reside external to the processor chipset (e.g., remote to the processor 800).

[0136] The memory 804 may store computer-readable, computer-executable code including instructions that, when executed by the processor 800, cause the processor 800 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such as system memory or another type of memory. The controller 802 and/or the processor 800 may be configured to execute computer-readable instructions stored in the memory 804 to cause the processor 800 to perform various functions. For example, the processor 800 and/or the controller 802 may be coupled with or to the memory 804, the processor 800, the controller 802, and the memory 804 may be configured to perform various functions described herein. In some examples, the processor 800 may include multiple processors and the memory 804 may include multiple memories. One or more of the multiple processors may be coupled with one or more of the multiple memories, which may, individually or collectively, be configured to perform various functions herein.

[0137] The one or more ALUs 806 may be configured to support various operations in accordance with examples as described herein. In some implementations, the one or more ALUs 806 may reside within or on a processor chipset (e.g., the processor 800). In some other implementations, the one or more ALUs 806 may reside external to the processor chipset (e.g., the processor 800). One or more ALUs 806 may perform one or more computations such as addition, subtraction, multiplication, and division on data. For example, one or more ALUs 806 may receive input operands and an operation code, which determines an operation to be executed. One or more ALUs 806 be configured with a variety of logical and arithmetic circuits, including adders, subtractors, shifters, and logic gates, to process and manipulate the data according to the operation. Additionally, or alternatively, the one or more ALUs 806 may support logical operations such as AND, OR, exclusive-OR (XOR), not-OR (NOR), and not- AND (NAND), enabling the one or more ALUs 806 to handle conditional operations, comparisons, and bitwise operations.

[0138] The processor 800 may support wireless communication in accordance with examples as disclosed herein. The processor 800 may be configured to or operable to support a means for receiving a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; obtaining the ML model based at least in part on the identifier of the ML model; determining whether a performance of the ML model satisfies the performance threshold associated with the ML model; and outputting a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold. In another example, the processor 800 may be configured to or operable to support a means for outputting a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; and receiving a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold. In another example, the processor 800 may be configured to or operable to support a means for receiving a request message for comparison data for evaluating the performance of an ML model used by the network entity; obtaining the requested comparison data; and transmitting a response message comprising the requested comparison data.

[0139] Figure 9 illustrates an example of a NE 900 in accordance with aspects of the present disclosure. The NE 900 may include a processor 902, a memory 904, a controller 906, and a transceiver 908. The processor 902, the memory 904, the controller 906, or the transceiver 908, or various combinations thereof or various components thereof may be examples of means for performing various aspects of the present disclosure as described herein. These components may be coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces.

[0140] The processor 902, the memory 904, the controller 906, or the transceiver 908, or various combinations or components thereof may be implemented in hardware (e.g., circuitry). The hardware may include a processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or other programmable logic device, or any combination thereof configured as or otherwise supporting a means for performing the functions described in the present disclosure.

[0141] The processor 902 may include an intelligent hardware device (e.g., a general- purpose processor, a DSP, a CPU, an ASIC, an FPGA, or any combination thereof). In some implementations, the processor 902 may be configured to operate the memory 904. In some other implementations, the memory 904 may be integrated into the processor 902. The processor 902 may be configured to execute computer-readable instructions stored in the memory 904 to cause the NE 900 to perform various functions of the present disclosure.

[0142] The memory 904 may include volatile or non-volatile memory. The memory 904 may store computer-readable, computer-executable code including instructions when executed by the processor 902 cause the NE 900 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such the memory 904 or another type of memory. Computer-readable media includes both non- transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium may be any available medium that may be accessed by a general-purpose or special-purpose computer.

[0143] In some implementations, the processor 902 and the memory 904 coupled with the processor 902 may be configured to cause the NE 900 to perform one or more of the functions described herein (e.g., executing, by the processor 902, instructions stored in the memory 904). For example, the processor 902 may support wireless communication at the NE 900 in accordance with examples as disclosed herein. The NE 900 may be configured to support a means for receiving a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; obtaining the ML model based at least in part on the identifier of the ML model; determining whether a performance of the ML model satisfies the performance threshold associated with the ML model; and outputting a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold. In another example, the NE 900 may be configured to or operable to support a means for outputting a request message comprising an identifier of an ML model and a performance threshold associated with the ML model; and receiving a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold. In another example, the NE 900 may be configured to or operable to support a means for receiving a request message for comparison data for evaluating the performance of an ML model used by the network entity; obtaining the requested comparison data; and transmitting a response message comprising the requested comparison data..

[0144] The controller 906 may manage input and output signals for the NE 900. The controller 906 may also manage peripherals not integrated into the NE 900. In some implementations, the controller 906 may utilize an operating system such as iOS®, ANDROID®, WINDOWS®, or other operating systems. In some implementations, the controller 906 may be implemented as part of the processor 902.

[0145] In some implementations, the NE 900 may include at least one transceiver 908. In some other implementations, the NE 900 may have more than one transceiver 908. The transceiver 908 may represent a wireless transceiver. The transceiver 908 may include one or more receiver chains 910, one or more transmitter chains 912, or a combination thereof.

[0146] A receiver chain 910 may be configured to receive signals (e.g., control information, data, packets) over a wireless medium. For example, the receiver chain 910 may include one or more antennas for receive the signal over the air or wireless medium. The receiver chain 910 may include at least one amplifier (e.g., a low-noise amplifier (LNA)) configured to amplify the received signal. The receiver chain 910 may include at least one demodulator configured to demodulate the receive signal and obtain the transmitted data by reversing the modulation technique applied during transmission of the signal. The receiver chain 910 may include at least one decoder for decoding the processing the demodulated signal to receive the transmitted data.

[0147] A transmitter chain 912 may be configured to generate and transmit signals (e.g., control information, data, packets). The transmitter chain 912 may include at least one modulator for modulating data onto a carrier signal, preparing the signal for transmission over a wireless medium. The at least one modulator may be configured to support one or more techniques such as amplitude modulation (AM), frequency modulation (FM), or digital modulation schemes like phase-shift keying (PSK) or quadrature amplitude modulation (QAM). The transmitter chain 912 may also include at least one power amplifier configured to amplify the modulated signal to an appropriate power level suitable for transmission over the wireless medium. The transmitter chain 912 may also include one or more antennas for transmitting the amplified signal into the air or wireless medium.

[0148] Figure 10 illustrates a flowchart of a method 1000 in accordance with aspects of the present disclosure. The operations of the method may be implemented by a network entity such as an LMF, as described herein. In some implementations, the network entity may execute a set of instructions to control the function elements of the network entity to perform the described functions. [0149] At 1002, the method may include receiving a request message comprising an identifier of an ML model and a performance threshold associated with the ML model.

[0150] The operations of 1002 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1002 may be performed by the LMF-T 502 as described with reference to Figures 5 and 6.

[0151] At 1004, the method may include obtaining the ML model based at least in part on the identifier of the ML model.

[0152] The operations of 1004 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1004 may be performed by the LMF-T 502 as described with reference to Figures 5 and 6.

[0153] At 1006, the method may include determining whether a performance of the ML model satisfies the performance threshold associated with the ML model.

[0154] The operations of 1006 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1006 may be performed the LMF-T 502 as described with reference to Figures 5 and 6.

[0155] At 1008, the method may include transmitting a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold.

[0156] The operations of 1008 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1008 may be performed by the LMF-T 502 as described with reference to Figures 5 and 6.

[0157] It should be noted that the method described herein describes A possible implementation, and that the operations and the steps may be rearranged or otherwise modified and that other implementations are possible.

[0158] Figure 11 illustrates a flowchart of a method 1100 in accordance with aspects of the present disclosure. The operations of the method may be implemented by a network entity such as an LMF, as described herein. In some implementations, the network entity may execute a set of instructions to control the function elements of the network entity to perform the described functions.

[0159] At 1102, the method may include transmitting a request message comprising an identifier of an ML model and a performance threshold associated with the ML model.

[0160] The operations of 1102 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1102 may be performed by the LMF-I 504 as described with reference to Figures 5 and 6.

[0161] At 1104, the method may include receiving a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold.

[0162] The operations of 1104 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1104 may be performed by the LMF-I 504 as described with reference to Figures 5 and 6.

[0163] It should be noted that the method described herein describes a possible implementation, and that the operations and the steps may be rearranged or otherwise modified and that other implementations are possible.

[0164] The description herein is provided to enable a person having ordinary skill in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to a person having ordinary skill in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not limited to the examples and designs described herein but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A network entity storing code comprising instructions executable to cause the network entity to: receive a request message comprising an identifier of a machine learning (ML) model and a performance threshold associated with the ML model; obtain the ML model based at least in part on the identifier of the ML model; determine whether a performance of the ML model satisfies the performance threshold associated with the ML model; and transmit a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold.

2. The network entity of claim 1 , wherein the instructions are further executable to cause the network entity to: obtain model data and comparison data; and compare the model data with the comparison data, wherein to determine whether the performance of the ML model satisfies the performance threshold associated with the ML model is based at least in part on the comparison of the model data with the comparison data.

3. The network entity of claim 2, wherein the model data comprises a current accuracy of a location determined using the ML model and the comparison data comprises a previous accuracy of a location previously determined by the ML model.

4. The network entity of claim 2 or 3, wherein the model data comprises a current accuracy of a location determined using the ML model and the comparison data comprises a previous accuracy of a location determined by other positioning methods.

5. The network entity of claim 3 or 4, wherein the instructions are further executable to cause the network entity to obtain the previous accuracy from one or more of an analytic data repository function (ADRF) entity or an operations, administration and maintenance (OAM) entity.

6. The network entity of any of claims 3 to 5, wherein the instructions are further executable to cause the network entity to obtain the accuracy of the location determined using the ML model from a second network entity.

7. The network entity of any preceding claim, wherein the model data comprises a location determined using the ML model and the comparison data comprises a known value of the location.

8. The network entity of claim 7, wherein the instructions are further executable to cause the network entity to obtain the known value of the location from one or more of: a user equipment (UE), a radio access network (RAN) node, or an operations, administration and maintenance (OAM) entity.

9. The network entity of any preceding claim, wherein the model data comprises a statistical parameter of training data of the ML model, and wherein the comparison data comprises a statistical parameter of corresponding current data.

10. The network entity of claim 9, wherein the instructions are further executable to cause the network entity to obtain the training data of the ML model from a database accessible to the network entity.

11. The network entity of any preceding claim, wherein the instructions are further executable to cause the network entity to obtain a replacement ML model based at least in part on the performance of the ML model failing to satisfy the performance threshold.

12. The network entity of claim 11, wherein the instructions are further executable to cause the network entity to obtain the replacement ML model by training the replacement ML model to satisfy the performance threshold.

13. The network entity of claim 11 or 12, wherein the response message comprises one or more of the replacement ML model or an indication of a network location where the replacement ML model can be accessed.

14. The network entity of any preceding claim, wherein the instructions are further executable to cause the network entity to determine whether the performance of the ML model satisfies the performance threshold associated with the ML model for at least a set of one or more time instances or during at least one duration, wherein the request message indicates the set of one or more time instances or the at least one duration.

15. A method performed by a network entity, the method comprising: receiving a request message comprising an identifier of a machine learning (ML) model and a performance threshold associated with the ML model; obtaining the ML model based at least in part on the identifier of the ML model; determining whether a performance of the ML model satisfies the performance threshold associated with the ML model; and transmitting a response message comprising an indication of whether the performance of the ML model satisfies the performance threshold.

16. A network entity storing code comprising instructions executable to cause the network entity to: transmit a request message comprising an identifier of a machine learning (ML) model and a performance threshold associated with the ML model; and receive a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold.

17. The network entity of claim 16, wherein the network entity uses the ML model to determine positions, wherein the instructions are further executable to cause the network entity to: receive a request message for comparison data; obtain the requested comparison data; and transmit a response message comprising the requested comparison data.

18. The network entity of claim 17, wherein the instructions are further executable to cause the network entity to obtain the requested comparison data by, each time the network entity uses the ML model to determine a position, recording the determined position and/or an accuracy of the determined position.

19. The network entity of claim 17, wherein the instructions are further executable to cause the network entity to obtain the requested comparison data by, each time the network entity determines a position, recording the determined position and/or an accuracy of the determined position.

20. A method performed by a network entity, the method comprising: transmitting a request message comprising an identifier of a machine learning (ML) model and a performance threshold associated with the ML model; and receiving a response message comprising an indication of whether a performance of the ML model satisfies the performance threshold.