EP4214912A1

EP4214912A1 - Non-realtime services for ai/ml

Info

Publication number: EP4214912A1
Application number: EP21870184.5A
Authority: EP
Inventors: Jaemin HAN; Qian Li; Leifeng RUAN; Geng Wu; Dawei YING
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 2020-09-16
Filing date: 2021-09-16
Publication date: 2023-07-26
Also published as: WO2022060923A1

Abstract

An apparatus for a non real-time (non-RT) radio access network intelligence controller (RIC) services for artificial intelligence (AI)/machine learning (ML) in an open radio access network (O-RAN), the apparatus including a processing circuitry configured to receive, from a Non-RT RIC application (rApp), an artificial intelligence (AI)/machine learning (ML) training service request, the AI/ML training service request including an indication of an AI/ML model structure, training preferences, and a training data description, and configured to send a training process identification (ID) to the rAPP, the training process ID identifying the AI/ML training service request. Additionally, the processing circuitry is configured to perform AI/ML model monitoring, AI/ML model management, and AI/ML model repository. The rAPP is configured to use the services provided by the Non-RT RIC.

Description

NON-REALTIME SERVICES FOR AI/ML

PRIORITY CLAIM

[0001] This application claims the benefit of priority to United States Provisional Patent Application 63/079,248, filed September 16, 2020, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

[0002] Aspects pertain to wireless communications. Some aspects relate to wireless networks including 3 GPP (Third Generation Partnership Project) networks, 3GPP LTE (Long Term Evolution) networks, 3GPP LTE-A (LTE Advanced) networks, (MulteFire, LTE-U), and fifth-generation (5G) networks including 5G new radio (NR) (or 5G-NR) networks, 5G networks such as 5G NR unlicensed spectrum (NR-U) networks and other unlicensed networks including Wi-Fi, CBRS (OnGo), etc. Other aspects are directed to Open RAN (O-RAN) architectures and, more specifically, techniques for providing services for artificial intelligence (AI) and machine learning (ML) in a non-real-time (Non-RT) radio access network (RAN) intelligent controller (RIC) (Non-RT RIC).

BACKGROUND

[0003] Mobile communications have evolved significantly from early voice systems to today’s highly sophisticated integrated communication platform.

With the increase in different types of devices communicating with various network devices, usage of 3GPP LTE systems has increased. The penetration of mobile devices (user equipment or UEs) in modem society has continued to drive demand for a wide variety of networked devices in many disparate environments. Fifth-generation (5G) wireless systems are forthcoming and are expected to enable even greater speed, connectivity, and usability. Next generation 5G networks are expected to increase throughput, coverage, and robustness and reduce latency and operational and capital expenditures. 5G new radio (5G-NR) networks will continue to evolve based on 3GPP LTE-Advanced with additional potential new radio access technologies (RATs) to enrich people’s lives with seamless wireless connectivity solutions delivering fast, rich content and services. As current cellular network frequency is saturated, higher frequencies, such as millimeter wave (mmWave) frequency, can be beneficial due to their high bandwidth. [0004] Potential LTE operation in the unlicensed spectrum includes (and is not limited to) the LTE operation in the unlicensed spectrum via dual connectivity (DC), or DC-based LAA, and the standalone LTE system in the unlicensed spectrum, according to which LTE-based technology solely operates in the unlicensed spectrum without requiring an “anchor” in the licensed spectrum, called MulteFire. MulteFire combines the performance benefits of LTE technology with the simplicity of Wi-Fi-like deployments.

[0005] Further enhanced operation of LTE and NR. systems in the licensed, as well as unlicensed spectrum, is expected in future releases and 5G systems such as O-RAN systems. Such enhanced operations can include techniques for AI and ML for O-RAN networks.

BRIEF DESCRIPTION OF THE FIGURES [0006] In the figures, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The figures illustrate generally, by way of example, but not by way of limitation, various aspects discussed in the present document.

[0007] FIG. 1 illustrates an example Open RAN (O-RAN) system architecture. [0008] FIG. 2 illustrates a logical architecture of the O-RAN system of FIG. 1. [0009] FIG. 3 illustrates a system for a Non-RT RIC, in accordance with some embodiments.

[0010] FIG. 4 illustrates a method for AI/ML training service requests, in accordance with some embodiments.

[0011] FIG. 5 illustrates a method for model training, in accordance with some embodiments. [0012] FIG. 6 illustrates a method for pushing an AI/ML model into an AI/ML model repository, in accordance with some embodiments.

[0013] FIG. 7 illustrates a method for pulling an AI/ML model from an AI/ML model repository, in accordance with some embodiments.

[0014] FIG. 8 illustrates a method for removing an AI/ML model from an AI/ML model repository, in accordance with some embodiments.

[0015] FIG. 9 illustrates a method for AI/ML model monitoring and management services, in accordance with some embodiments.

[0016] FIG. 10 illustrates a request to resume AI/ML monitoring, in accordance with some embodiments. [0017] FIG. 11 illustrates a method to request to delete an ongoing or suspended monitor service, in accordance with some embodiments.

DETAILED DESCRIPTION

[0018] The following description and the drawings sufficiently illustrate aspects to enable those skilled in the art to practice them. Other aspects may incorporate structural, logical, electrical, process, and other changes. Portions and features of some aspects may be included in or substituted for, those of other aspects.

Aspects outlined in the claims encompass all available equivalents of those claims.

[0019] FIG. 1 provides a high-level view of an Open RAN (O-RAN) architecture 100. The O-RAN architecture 100 includes four O-RAN defined interfaces - namely, the A1 interface, the O1 interface, the O2 interface, and the Open Fronthaul Management (M)-plane interface - which connect the Service Management and Orchestration (SMO) framework 102 to O-RAN network functions (NFs) 104 and the O-Cloud 106. The SMO 102 (described in Reference [R13]) also connects with an external system 110, which provides enrichment data to the SMO 102. FIG. 1 also illustrates that the A1 interface terminates at an O- RAN Non-Real Time (RT) RAN Intelligent Controller (RIC) 112 in or at the SMO 102 and at the O-RAN Near-RT RIC 114 in or at the O-RAN NFs 104. The O- RAN NFs 104 can be virtual network functions (VNFs) such as virtual machines (VMs) or containers, sitting above the O-Cloud 106 and/or Physical Network Functions (PNFs) utilizing customized hardware. All O-RAN NFs 104 are expected to support the O1 interface when interfacing with the SMO framework 102. The O-RAN NFs 104 connect to the NG-Core 108 via the NG interface (which is a 3 GPP defined interface). The Open Fronthaul M-plane interface between the SMO 102 and the O-RAN Radio Unit (O-RU) 116 supports the O- RU 116 management in the O-RAN hybrid model as specified in Reference [R16].

The Open Fronthaul M-plane interface is an optional interface to the SMO 102 that is included for backward compatibility purposes as per Reference [R16] and is intended for management of the O-RU 116 in hybrid mode only. The management architecture of flat mode (see Reference [R12]) and its relation to the O1 interface for the O-RU 116 is in development. The O-RU 116 termination of the O1 interface towards the SMO 102 as specified in Reference [R12].

[0020] FIG. 2 shows an O-RAN logical architecture 200 corresponding to the O- RAN architecture 100 of FIG. 1. In FIG. 2, the SMO 202 corresponds to the SMO 102, O-Cloud 206 corresponds to the O-Cloud 106, the non-RT RIC 212 corresponds to the non-RT RIC 112, the near-RT RIC 214 corresponds to the near- RT RIC 114, and the O-RU 216 corresponds to the O-RU 116 of FIG. 2, respectively. The O-RAN logical architecture 200 includes a radio portion and a management portion. [0021] The management portion/side of the architectures 200 includes the SMO

Framework 202 containing the non-RT RIC 212, and may include the O-Cloud 206. The O-Cloud 206 is a cloud computing platform including a collection of physical infrastructure nodes to host the relevant O-RAN functions (e.g., the near- RT RIC 214, O-RAN Central Unit-Control Plane (O-CU-CP) 221, O-RAN Central Unit-User Plane O-CU-UP 222, and the O-RAN Distributed Unit (O-DU) 215, supporting software components (e.g., OSs, VMMs, container runtime engines, ML engines, etc.), and appropriate management and orchestration functions.

[0022] The radio portion/side of the logical architecture 200 includes the near-RT RIC 214, the O-DU 215, the O-RAN Radio Unit (O-RU) 216, the O-CU-CP 221, and the O-CU-UP 222 functions. The radio portion/side of the logical architecture 200 may also include the O-e/gNB 210.

[0023] The O-DU 215 is a logical node hosting Radio Link Control (RLC), media access control (MAC), and higher physical (PHY) layer entities/elements (High- PHY layers) based on a lower layer functional split. The O-RU 216 is a logical node hosting lower PHY layer entities/elements (Low-PHY layer) (e.g., FFT/iFFT, PRACH extraction, etc.) and RF processing elements based on a lower layer functional split. Virtualization of O-RU 216 is FFS. The O-CU-CP 221 is a logical node hosting the RRC and the control plane (CP) part of the PDCP protocol. The O-CU-UP 222 is a logical node hosting the user plane part of the PDCP protocol and the SDAP protocol.

[0024] An E2 interface terminates at a plurality of E2 nodes. The E2 nodes are logical nodes/entities that terminate the E2 interface. For NR/5G access, the E2 nodes include the O-CU-CP 221, O-CU-UP 222, O-DU 215, or any combination of elements as defined in Reference [R15], For E-UTRA access the E2 nodes include the O-e/gNB 210. As shown in FIG. 2, the E2 interface also connects the O-e/gNB 210 to the Near-RT RIC 214. The protocols over E2 interface are based exclusively on Control Plane (CP) protocols. The E2 functions are grouped into the following categories: (a) near-RT RIC 214 services (REPORT, INSERT, CONTROL and POLICY, as described in Reference [R15]); and (b) near-RT RIC 214 support functions, which include E2 Interface Management (E2 Setup, E2 Reset, Reporting of General Error Situations, etc.) and Near-RT RIC Service Update (e.g., capability exchange related to the list of E2 Node functions exposed over E2).

[0025] FIG. 2 shows the Uu interface between a UE 201 and O-e/gNB 210 as well as between the UE 201 and O-RAN components. The Uu interface is a 3 GPP defined interface (see e.g., sections 5.2 and 5.3 of Reference [R07]), which includes a complete protocol stack from LI to L3 and terminates in the NG-RAN or E-UTRAN. The O-e/gNB 210 is an LTE eNB (see Reference [R04]), a 5G gNB or ng-eNB (see Reference [R06]) that supports the E2 interface. The O-e/gNB 210 may be the same or similar as discussed in FIGS. 3-11. The UE 201 may correspond to UEs discussed with respect to FIGS. 3-11 and/or the like. There may be multiple UEs 201 and/or multiple O-e/gNB 210, each of which may be connected to one another the via respective Uu interfaces. Although not shown in FIG. 2, the O-e/gNB 210 supports O-DU 215 and O-RU 216 functions with an Open Fronthaul interface between them.

[0026] The Open Fronthaul (OF) interfaced) is/are between O-DU 215 and O- RU 216 functions (see References [R16] and [R17].) The OF interfaced) includes the Control User Synchronization (CUS) Plane and Management (M) Plane. FIGS. 1 and 2 also show that the O-RU 216 terminates the OF M-Plane interface towards the O-DU 215 and optionally towards the SMO 202 as specified in Reference [R16]. The O-RU 216 terminates the OF CUS-Plane interface towards the O-DU 215 and the SMO 202. [0027] The F1-c interface connects the O-CU-CP 221 with the O-DU 215. As defined by 3 GPP, the F1-c interface is between the gNB-CU-CP and gNB-DU nodes (see References [R07] and [R10].) However, for purposes of O-RAN, the F1-c interface is adopted between the O-CU-CP 221 with the O-DU 215 functions while reusing the principles and protocol stack defined by 3 GPP and the definition of interoperability profile specifications.

[0028] The F1-u interface connects the O-CU-UP 222 with the O-DU 215. As defined by 3 GPP, the F1-u interface is between the gNB-CU-UP and gNB-DU nodes (see References [R07] and [R10]). However, for purposes of O-RAN, the F1-u interface is adopted between the O-CU-UP 222 with the O-DU 215 functions while reusing the principles and protocol stack defined by 3GPP and the definition of interoperability profile specifications.

[0029] The NG-c interface is defined by 3GPP as an interface between the gNB- CU-CP and the AMF in the 5GC (see Reference [R06]). The NG-c is also referred as the N2 interface (see Reference [R06]). The NG-u interface is defined by 3GPP, as an interface between the gNB-CU-UP and the UPF in the 5GC (see Reference [R06]). The NG-u interface is referred as the N3 interface (see Reference [R06]). In O-RAN, NG-c and NG-u protocol stacks defined by 3GPP are reused and may be adapted for O-RAN purposes. [0030] The X2-c interface is defined in 3GPP for transmitting control plane information between eNBs or between eNB and en-gNB in EN-DC. The X2-u interface is defined in 3GPP for transmitting user plane information between eNBs or between eNB and en-gNB in EN-DC (see e.g., [005], [006]). In O-RAN, X2- c and X2-u protocol stacks defined by 3GPP are reused and may be adapted for O-RAN purposes.

[0031] The Xn-c interface is defined in 3GPP for transmitting control plane information between gNBs, ng-eNBs, or between an ng-eNB and gNB. The Xn-u interface is defined in 3GPP for transmitting user plane information between gNBs, ng-eNBs, or between ng-eNB and gNB (see e.g., References [R06] and [R08]). In O-RAN, Xn-c and Xn-u protocol stacks defined by 3GPP are reused and may be adapted for O-RAN purposes

[0032] The E1 interface is defined by 3GPP as being an interface between the gNB-CU-CP (e.g., gNB-CU-CP 3728) and gNB-CU-UP (see e.g., [007], [009]). In O-RAN, E1 protocol stacks defined by 3GPP are reused and adapted as being an interface between the O-CU-CP 221 and the O-CU-UP 222 functions.

[0033] The O-RAN Non-Real Time (RT) RAN Intelligent Controller (RIC) 212 is a logical function within the SMO framework 102, 202 that enables non-real- time control and optimization of RAN elements and resources; AI/machine learning (ML) workflow(s) including model training, inferences, and updates; and policy-based guidance of applications/features in the Near-RT RIC 214.

[0034] The O-RAN near-RT RIC 214 is a logical function that enables near-real- time control and optimization of RAN elements and resources via fine-grained data collection and actions over the E2 interface. The near-RT RIC 214 may include one or more AI/ML workflows including model training, inferences, and updates.

[0035] The non-RT RIC 212 can be an ML training host to host the training of one or more ML models. The ML data can collected from one or more of the following: the Near-RT RIC 214, O-CU-CP 221, O-CU-UP 222, O-DU 215, O- RU 216, external enrichment source 110 of FIG. 1, and so forth. For supervised learning, and the ML training host and/or ML inference host/actor can be part of the non-RT RIC 212 and/or the near-RT RIC 214. For unsupervised learning, the ML training host and ML inference host/actor can be part of the non-RT RIC 212 and/or the near-RT RIC 214. For reinforcement learning, the ML training host and ML inference host/actor are co-located as part of the near-RT RIC 214. In some implementations, the non-RT RIC 212 may request or trigger ML model training in the training hosts regardless of where the model is deployed and executed. ML models may be trained and not currently deployed.

[0036] In some implementations, the non-RT RIC 212 provides a query-able catalog for an ML designer/developer to publish/install trained ML models (e.g., executable software components). In these implementations, the non-RT RIC 212 may provide discovery mechanism if a particular ML model can be executed in a target ML inference host (MF), and what number and type of ML models can be executed in the target ML inference host. The Near-RT RIC 214 is a managed function (MF). For example, there may be three types of ML catalogs made discoverable by the non-RT RIC 212: a design-time catalog (e.g., residing outside the non-RT RIC 212 and hosted by some other ML platform(s)), a training/deployment-time catalog (e.g., residing inside the non-RT RIC 212), and a run-time catalog (e.g., residing inside the non-RT RIC 212). The non-RT RIC 212 supports necessary capabilities for ML model inference in support of ML assisted solutions running in the non-RT RIC 212 or some other ML inference host. These capabilities enable executable software to be installed such as VMs, containers, etc. The non-RT RIC 212 may also include and/or operate one or more ML engines, which are packaged software executable libraries that provide methods, routines, data types, etc., used to run ML models. The non-RT RIC 212 may also implement policies to switch and activate ML model instances under different operating conditions.

[0037] The non-RT RIC 22 is able to access feedback data (e.g., FM, PM, and network KPI statistics) over the O1 interface on ML model performance and perform necessary evaluations. If the ML model fails during runtime, an alarm can be generated as feedback to the non-RT RIC 212. How well the ML model is performing in terms of prediction accuracy or other operating statistics it produces can also be sent to the non-RT RIC 212 over O1. The non-RT RIC 212 can also scale ML model instances running in a target MF over the O1 interface by observing resource utilization in MF. The environment where the ML model instance is running (e.g., the MF) monitors resource utilization of the running ML model. This can be done, for example, using an ORAN-SC component called ResourceMonitor in the near-RT RIC 214 and/or in the non-RT RIC 212, which continuously monitors resource utilization. If resources are low or fall below a certain threshold, the runtime environment in the near-RT RIC 214 and/or the non- RT RIC 212 provides a scaling mechanism to add more ML instances. The scaling mechanism may include a scaling factor such as an number, percentage, and/or other like data used to scale up/down the number of ML instances. ML model instances running in the target ML inference hosts may be automatically scaled by observing resource utilization in the MF. For example, the Kubemetes® (K8s) runtime environment typically provides an auto-scaling feature.

[0038] The A1 interface is between the non-RT RIC 212, which is within - the SMO 202) and the near-RT RIC 214. The A1 interface supports three types of services as defined in Reference [R14], including a Policy Management Service, an Enrichment Information Service, and ML Model Management Service. A1 policies have the following characteristics compared to persistent configuration as defined in Reference [R14]: A1 policies are not critical to traffic; A1 policies have temporary validity; A1 policies may handle individual UE or dynamically defined groups of UEs; A1 policies act within and take precedence over the configuration; and A1 policies are non-persistent, i.e., do not survive a restart of the near-RT RIC.

[0039] A technical problem is how to train and maintain good AI/ML models to be used by an inference host to perform E2 control and other controls. The disclosed examples address this issue by including two training hosts: one in the non-RT RIC and one in the near-RT RIC. The training host in the near-RT RIC performs online learning, which may use different data than the offline learning, and ensures that an adequate AI/ML model is being used by the inference host. The training host of the non-RT RIC performs offline learning and transfers an initial model and updated models to a model repository that is used to store AI/ML model that may be used by the training host of the near-RT RIC. [0040] A deployment is disclosed of online reinforcement learning in the Near- RT RIC. The AI/ML training host and inference host are located in the Near-RT RIC, while an offline learning host and ML model repository reside in the Non- RT RIC. The deployment reduces communication and feedback delay between the ML training host and the ML inference host. The delay reduction is essential for online reinforcement learning, especially for generating fast changing decision-making policies that adapt to highly dynamic environments. The ML model repository ensures performance of the online reinforcement learning by saving the most accurate and best performing ML models. [0041] Examples disclose a deployment scenario for online reinforcement learning in the Near-RT RIC, which incorporates an online training host and inference host in the Near-RT RIC, while an offline training host and ML model repository reside in SMO/Non-RT RIC.

[0042] A technical problem is how to train and monitor AI/ML models that are used within O-RAN architecture. Examples address this technical problem by providing AI/ML model training services and model monitoring service in a Non-RT RIC to enable Non-RT RIC applications (rApps) to operate. rApps are applications deployed in the Non-RT RIC. The monitoring services and management services in the Non-RT RIC are exposed to the rApps via the R1 interface. Additionally, a model repository is provided in the Non-RT RIC. [0043] FIG. 3 illustrates a system 300 for a Non-RT RIC, in accordance with some embodiments. The arrows with circles, e.g., O2 between O2 termination 336 and O2 cloud 356, are O-RAN defined interfaces. The regular arrows, e.g.,

Human-Machine interface 360, except SMO internal interface 366, are external interfaces. SMO internal interface 366 is a proprietary interface, e.g., outside O- RAN specification scope.

[0044] The rApps 301 include rApp 1 302.1, rApp 2302.2, ..., rApp N 302.N. The R1 310 interface is between rApps 301 and Non-RT RIC framework 308 which is used to support third-party applications deployed in the Non-RT RIC 306 for open and intelligent RAN operation. rApps 301 are modular applications that leverage the functionality exposed by the Non-RT RIC 306 to provide added value services, including AI/ML-assisted solutions. The R1 interface 310 collects performance measurements and provide management instructions for rApps 301. rApps 301 use the AI/ML services provided by the Non-RT RIC framework 308 for AI/ML training, monitoring, and management. [0045] Non-RT RIC 306 AI/ML services are provided to rApps 301. Non-RT RIC framework provides one or more of the following AI/ML services over the

R1 310 interface: AI/ML model training; AI/ML model monitoring; andZor, AI/ML model management. AI/ML service function, which is an inherent function of Non-RT RIC, trains, monitors, and manages AI/ML models in rApps 301. In addition, an AI/ML model repository is provided to store trained AI/ML models in Non-RT RIC.

[0046] The SMO framework 304, the Non-RT RIC framework functionality 312, the SMO service exposure function 314, the rAPP service exposure functions 316, the Non-RT RIC framework service exposure function 318, the other Non-RT RIC framework functions 320, the rAPP management functions 322, the A1 policy functions 324, the A1 E1 functions 326, the A1 ML functions

328, the A1 termination 330, the Near-RT RIC 332, the E2 nodes 334, the A1 335 interface, the O2 termination 336, the O1 termination 338, the Implementation variability 340, the Function 1 342.1, function 2342.2, ..., function N 342.N, the external E1 termination 344, the external AI/ML termination 346, the human-machine termination 348, external AI/ML servers 350, the external E1 sources 352, the RAN intent injection 354, the O2 cloud 356, the inherent SMO framework functionality 358, the other SMO framework functions 359, the Human-Machine interface 360, the external AI/ML interface 362, the external enrichment information (El) interface 364, and the SMO internal interface 366 are as disclosed herein and in the References.

[0047] FIG. 4 illustrates a method 400 for AI/ML training service requests, in accordance with some embodiments. FIG. 4 illustrates a method 400 for an rApp 301 to request an AI/ML model training service. The rApp 301 describes the training data. For example, the method 400 may begin at operation 406 (Step 1) with the AI/ML training service request 407 being sent from an rApp 301 to the non-RT RIC 306 where the AI/ML training service request 407 includes one or more of the following: training data description 405, AI/ML model structure 409, training preferences 411, and so forth. The training data description 405 includes one or more of the following: data type, sampling rate, period, granularity, and/or conditions.

[0048] AI/ML model training services are provided by the Non-RT RIC framework functionality 312. To utilize the Non-RT RIC’s model training service, rApp 301 provides one or more of the following: a training data description 405, an AI/ML model structure 409, and training preferences and requirements 411. Training data 413 that fits the training data description 405 can be collected from the Near-RT RIC 306 or E2 nodes 334 via the O1 interface, and the training data can be collected as the output from other rApps 301. The training data description 405 specifies the data type, sampling rate, sampling period, granularity, and so forth. For example, if an AI/ML model is used to predict a UE’s RF signal strength, e.g., SS-RSRP, based on the previous measurements, then the training data description for this model may be per-UE SS-RSRP measurements for every slot within last 100 ms. For example: data type: SS-RSRP; sampling rate: every slot; sampling period: 100 ms; granularity: per-UE. Additional conditions may be attached to the training data description 405, for example, the above model can specify that only cell-edge UE’s SS- RSRP measurements are collected for training. The rApp 391 may make this request because the predicted UE signal strength is mainly used for HO optimization.

[0049] If there is no error, then the method 400 continues at operation 408 (Step 2a) with the AI/ML training service response, which includes one ore more of the following: training process identification (ID) 413 and/or an indication that the AI/ML training service request 406 was accepted. The Non-RT RIC accepts the request if it have access to the requested training data 413 (Step 2a), otherwise it rejects the training request at operation 410 (Step 2b).

[0050] If the AI/ML training service request 407 cannot be accommodated, then the method 400 continues at operation 410 (Step 2b) with the Non-RT RIC 306 sending the AI/ML training service response, which includes an error code 415 that may indicate that the AI/ML training service request 406 was rejected.

[0051] The Non-RT RIC 306 accepts the request if it has access to the requested training data 413 (Step 2a), and can accommodate the service request, otherwise it rejects the training request (Step 2b) at AI/ML training service response 410. [0052] The error code 415 may indicate one or more error cases, which includes one or more of the following: unknown data type, missing elements in the description (e.g., sampling rate, period, granularity, etc.), error in configuration (e.g., too fast sampling rate, too long sampling period, or too fine granularity, etc.), and/or other reasons. A training process ID 413 is assigned by the Non-RT RIC 306 if the request is accepted, and the unique training process ID 413 serves as a reference for the following steps for this model training.

[0053] FIG. 5 illustrates a method 500 for model training, in accordance with some embodiments. If the training data 413 is not available at the Non-RT RIC 306, then there is no need for the rApp 301 to perform method 500, expose AI/ML model structure to Non-RT RIC, or list the training requirements.

[0054] The method 500 begins at operation 504 (or Step 1) with the rApp 301 sending an AI/ML model exposure message with data 505. The data 504 includes one or more of the following: training process ID 413, model structure/configuration, pre-process, and labeling method, and so forth.

[0055] If the training request is accepted by the Non-RT RIC 306, rApp 301 shares the AI/ML model structure, e.g. data 505, with Non-RT RIC 306. An AI/ML model structure is an untrained model. An rApp 301 may build its own model or use pre-set models (with proper configurations) provided by the Non- RT RIC 301 platform. If data pre-processing is required before model training, such pre-processing methods needs to be exposed to the Non-RT RIC 301 in data 505 or in an earlier message. For supervised learning, the labelling method is required to be shared with the Non-RT RIC 301. Labelling can be regarded as a part of pre-processing. [0056] The method 500 continues at operation 506 (Step 2) with training configurations and/or requirements being sent from the rApp 301 to the Non-RT RIC 306. The rApp 301 is needs to configure the training options and/or provide requirements or goals/thresholds for training results. The data 507 includes configurable training options such as one or more of the following: loss function, optimizer, number of epochs, number of iterations/batches, learning rate, split ratio of training and validation data, gradient clipping, testing data set, and/or other options. If the rApp 301 does not configure one or more options in the above list, the Non-RT RIC 306 may use default configuration/value where the [0057] Non-RT RIC 306 can adjust these specified options during the training process in order to satisfy the goal/thresholds/requirement for the training results specified in the data 507 by the rApp 301.

[0058] If rApp 301 provides testing data set, e.g., in data 507, to the Non-RT RIC 306, then it should be consistent with specified training data, i.e., same data type. Training data and validation data are collected from RAN, and test data is provided by the rApp 301 independent from the training data 413, which may include validation data. In some embodiments, the terms validation data and test data may be switched in meaning. The data 507 may include training requirements, which includes one or more of the following: training loss requirements, metrics, validation requirements based on defined metrics, testing requirements based on defined metrics, and/or other requirements.

[0059] If an rApp 301 wants to make sure the training results satisfy certain requirements, it provides the definition of metrics of the metrics to use to determine if the AI/ML model is trained in the data 507. For example, a model for classification problem usually uses accuracy as metric, while a model for regression uses MSE (mean square error) as a common metric.

[0060] The method 500 continues at operation 514 (Step 3) with collect training data, which includes data 515. The Non-RT RIC 306 collects training data from the O-RAN 502. The training data may be collected at a different time such as before operation 504.

[0061] The method 500 continues at operation 516 (Step 4) with AI/ML training, which may include data 517. After receiving the model structure and training configurations from rApp 301 in the data 507 and collecting training data from RAN at operation 514, the Non-RT RIC 306 performs AI/ML training. [0062] Note that the training data collection (operation 514 or Step 3) can start right after accepting the training request, which can be earlier than model exposure such as is disclosed in FIG. 4.

[0063] If the training is successful, then the method 500 continues at operation 508 (Step 5a) with AI/ML model deployment, which may include data 509 that includes one or more of training process ID 413, trained model, and so forth. [0064] If the training fails, then the method 500 continues at operation 510 with AI/ML model training error (Step 5b), which may include data 511 that includes a training process ID 413 and one or more error codes. The error cases that are indicated by the error codes include one or more of the following: unknown training processing ID, invalid model structure/configuration, invalid training configuration, failed to satisfy training requirements, training consumes too many resources or takes too much time, and/or other reasons. [0065] The method 500 may, optionally, continue at operation 512 (Step 6) with the rApp sending training configurations and/or requirements to the Non-RT RIC 306. rApps 301 can request multiple rounds/attempts of training with different configurations or requirements (Step 6).

[0066] FIG. 6 illustrates a method 600 for pushing an AI/ML model into an AI/ML model repository, in accordance with some embodiments. The AI/ML model repository 601 is used for rApps 301 to store trained AI/ML models. One AI/ML model structure can have multiple copies of trained AI/ML models recorded in the model repository, e.g., with different sets of weights and AI/ML model structure information. The AI/ML model repository enables the rApp 301 to store and retrieve AI/ML models rather than updating existing trained model or training new models.

[0067] Apps 301 pushes an updated model and receives another model copy identifier/address for the new one from the AI/ML model repository 601 of the Non-RT RIC 306. If there is no need to keep the copy of the old version model, rApp 301 removes it from the repository. The AI/ML model repository 601 may provide benefits as follows. First, when the running model in an rApp 301 crushes, drifts too much, or simply malfunctions, then the AI/ML model repository 601 may provide the stored AI/ML model as a backup, which is supposed to be stable or may have been determined to be stable by the rApp 301 prior to storage in the AI/ML model repository 602. Second, when a new version of an rApp 301 is available, then it does not need to undergo the entire model training/verification procedure again, if its AI/ML model structure does not change. The trained model copy in the repository can be downloaded to the updated rApp 301. Third, an rApp 301 may share a trained AI/ML model with other trusted rApps 301 via the model repository. For example, an rApp 301 may share an identifier or address of the AI/ML model in the AI/ML model repository with its solution provider so that other trusted rApps 301 from the same solution provider can access/reuse this trained model. [0068] The method 600 begins at operation 606 (Step 1) with rApp 301 sending a push model to repository request, which includes data 607, to the Non- RT RIC 306. The data 607 includes the trained AI/ML model and, optionally, the allowed list of rApps 301, which are trusted and have access to the stored AI/ML model copy. In one example, the list of trusted rApps 301 is a list of rApp IDs. In another example, the list of allowed rApps is a set of filtering conditions, e.g., “rApp vendor = xxx && rApp version number > yy”.

[0069] If the AI/ML model pushing into the AI/ML model respositry 601 is successful, then the method 600 (Step 2a) continues at operation 608 with response to model push with data 609 that includes an address (or an identifier) for the model copy, which may indicate a successful push or this may be indicated by other data such as a field to return a successful push code.

[0070] If pushing fails, then the method 600 continues at operation 610 (Step 2b) which includes data 611 that includes error codes from the Non-RT RIC 306 and/or a model repository module. The error cases indicated by the error codes include one or more of the following: not enough memory resource and/or other reasons.

[0071] FIG. 7 illustrates a method 700 for pulling an AI/ML model from an AI/ML model repository, in accordance with some embodiments. The method 700 begins at operation 706 (Step 1) with rApp 301 sending a pull model from repository request, which includes data 707, to the Non-RT RIC 306. The data 707 includes one or more of a model copy ID/address, and rApp ID.

[0072] If pulling is successful, then the method 700 continues at operation 708 (Step 2a) with the Non-RT RIC 306 (or model repository) sending a response to model pull, which includes data 709, to the rApp 301. The data 709 includes the model copy.

[0073] If pulling fails, then the method 700 continues at operation 710 (Step 2b) with the Non-RT RIC 306 (or model repository) sending a response to model pull with data 711 where the data 711 indicates error codes. The error cases indicated by the error codes include one or more of the following: invalid model copy id/address, no access to the specified model copy, and/or other reasons. [0074] FIG. 8 illustrates a method 800 for removing an AI/ML model from an AI/ML model repository, in accordance with some embodiments. The method 800 begins at operation 806 (Step 1) with an rApp 301 sending a remove model from repository request, which includes data 807, to the non-RT RIC 306 (or model repository). The data 807 may include model copy ID/address, rApp ID, an indication that this is a model removal request, and so forth.

[0075] If removal of the model is successful, then the method 800 continues at operation 808 (or Step 2a) with the Non-RT RIC 306 (or model repository) sending a response to model remove, which includes data 809. The data 809 includes an indication of model removal success.

[0076] If removing fails, then the method 800 continues at operation 810 with the Non-RT RIC 306 (or model repository) sending a response to model remove, which includes data 811. The data 811 includes error codes. The error codes indicate one or more error cases such as invalid model copy id/address (e.g., model copy already removed), no access to the specified model copy, and/or other reasons.

[0077] FIG. 9 illustrates a method 900 for AI/ML model monitoring and management services, in accordance with some embodiments. The method 900 begins at operation 902 (Step 1) with the rApp 301 sending an AI/ML monitoring service request, which includes data 903, to the Non-RT RIC 306. The data 903 may include monitoring data description, monitoring metrics and configurations, retraining/termination conditions, and/or notification destination. For example, if an rApp 301 is not capable of (or does not wish to) to monitor and manage its internal AI/ML model(s), then may rely on the AI/ML model monitoring and management service in Non-RT RIC 306. The rApp 301 is provides monitoring data description, monitoring metrics and configuration, conditions for model retraining and termination, and model management notification destination to the Non-RT RIC 306 for model monitoring service. Note that rApp 301 does not need to expose its AI/ML model structure to the Non-RT RIC 306 for monitoring service.

[0078] The performance of the AI/ML model is evaluated by the Non-RT RIC 306 based on the monitoring data, which consists two parts: inferred model output from the rApp 301 and observed data from O-RAN 502 (Near-RT RIC 332 or E2 nodes 334). In the monitoring service request in operation 902 (Step 1), rApp may describe RAN data to be observed for monitoring in the same format as to the training data description, i.e., data type, sampling rate, period, granularity and optionally filtering conditions. The monitoring metrics are used to evaluate the performance of model inference, and the rApp configures the monitoring frequency. The conditions for model retraining and termination are used for model management, and they are constructed based on the defined metrics. The notification destination informs the Non-RT RIC 306 where to send the management notifications.

[0079] If the service request is accepted, the method 900 continues at operation 904 (Step 2a) with the Non-RT RIC 306 sending an AI/ML monitoring service response, which includes data 905. The data 905 includes model instance ID, which indicates the model is accepted and is a model instance ID assigned by the AI/ML service function of the Non-RT RIC 306 for model monitoring and management. (Note that model instance ID and training process ID, which is assigned by the training service, are not or need not be the same.) [0080] If the service request is not accepted, then the method 900 continues at operation 906 with the Non-RT RIC 306 (or AI/ML service function) sending an AI/ML monitoring service response, which includes data 907. The data 907 includes error codes such as request denied. The error codes may indicate error cases such as unknown monitoring data type, error in monitoring data configuration (e.g., missing components, invalid configurations, etc.), error in monitoring configuration (e.g., too frequent monitoring occasions), undefined monitoring metrics, invalid notification destination, and/or other reasons. The model instance ID is the unique ID for model monitoring and management. [0081] If the monitoring service is accepted, the method 900 continues. The method 900 may continue with operation 908 (Step 3a) with the Non-RT RIC 306 collecting observation data from Near-RT RIC 332 or E2 nodes 334 via O1 interface based on the data 903 that includes monitoring data description from rApp 301.

[0082] The method 900 continues at operation 910 (Step 3b) with the rApp 301 sending collected inferred data, which includes data 911, to the Non-RT RIC 306. The data 911 includes inferred data collected from rApp 301 via the R1 interface 310.

[0083] The method 900 continues at operation 912 (Step 4) with the Non-RT RIC 306 performing AI/ML model evaluation, which includes data 913. The Non-RT RIC 306 performs AI/ML model performance evaluation by computing the monitoring metrics using the observed data and inferred data. The rApp 306 may be responsibility for ensuring that the inferred data matches the observed data for meaningful metrics computing from RAN.

[0084] If there are errors during model performance evaluation, then the method 900 continues at operation 914 (Step 5a) with the Non-RT RIC 306 sending an AI/ML management notification to the rApp 301, which includes data 915. The data 915 may include model instance ID, error codes such as monitoring errors, and so forth. In this case, the monitoring service may be terminated and the rApp 301 can send another monitoring service request, e.g., operation 902, to the Non-RT RIC 301. The error codes indicate one or more of the following error cases: error in metric computation, unmatched observed and inferred data (e.g., unmatched dimension), suspension timer expires, and other reasons. Otherwise, based on the monitoring results, the Non-RT RIC 301 can manage the AI/ML model in rApps 301.

[0085] If the AI/ML model retraining condition is met, then the method 900 continues at operation 916 (Step 5b), which includes data 917, where the Non- RT RIC 301 sends a management notification to rApp to initiate another training process.

[0086] If the model termination condition is met, then the method 900 continues at operation 918 (Step 5c) with the Non-RT RIC 306 sending a management notification, which includes data 919, to rApp 301 to terminate the running model. The monitoring and management services are suspended by Non-RT RIC 306, if a management notification of model retraining or termination is sent to rApp 301. rApp can request another training service or retrieve a backup model copy form model repository. [0087] FIG. 10 illustrates a request to resume AI/ML monitoring, in accordance with some embodiments. The method 1000 begins at operation 1006 with the rApp 301 sending a AI/ML monitoring resume request, which includes data 1007, to the Non-RT RIC 306. The data 1007 includes a model instance ID. For example, after the AI/ML model gets updated (retrained or switched back to previous version), rApp 301 requests Non-RT RIC 306 to resume the monitoring service with model instance ID.

[0088] If the request to resume AI/ML monitoring is accepted, then the method 1000 continues at operation 1008 (Step 2a) with response to service resume, which includes data 1009. Data 1009 includes model instance ID, which may indicate that the request to resume monitoring has been accepted. [0089] If the request to resume AI/ML monitoring is not accepted, then the method 1000 continues at operation 1010 (Step 2b) with the Non-RT RIC 306 sending a response to service resume, which includes data 1011, to the rApp 301. The data 1011 includes error codes that indicate error cases such as invalid model instance ID, and/or other reasons.

[0090] The Non-RT RIC 306 maintains a timer for AI/ML monitoring service suspension. If the timer expires, then the Non-RT RIC terminates the monitoring service.

[0091] FIG. 11 illustrates a method 1100 to request to delete an ongoing or suspended monitor service, in accordance with some embodiments. The method 1100 begins at operation 1106 (Step 1) with AI/ML model monitoring delete request, which includes data 1107, being sent from the rApp 301 to the Non-RT RIC 306. The data 1107 may include model instance ID.

[0092] If the deletion is successful, then the method 1100 continues at operation 1108 (Step 2a), with response to service deletion, which includes data 1109, being sent from the Non-RT RIC 306 to the rApp 301. The data 1109 may include no content, which may indicate a successful deletion. The message as in all messages disclosed in conjunction with the figures may include an indication of the type of message such a type to indicate a response to service deletion. [0093] If the deletion is not successful, the method 1100 continues at operation 1110 (Step 2b) with response to service deletion, which may include data 1111. Data 1111 includes error codes that may indicate deletion failed, invalid model instance ID, and/or other reasons.

[0094] The methods described in conjunction with FIGS. 4-11 may include one or more additional operations. The operations of the methods described in conjunction with FIGS. 4-11 may be performed in a different order. One or more of the operations of the methods described in conjunction with FIGS. 4-11 may be optional.

[0095] REFERENCES

[0096] [R04] 3 GPP TS 36.401 v15.1.0 (2019-01-09).

[0097] [R05] 3 GPP TS 36.420 v15.2.0 (2020-01-09). [0098] [R06] 3 GPP TS 38.300 v16.0.0 (2020-01-08).

[0099] [R07] 3 GPP TS 38.401 v16.0.0 (2020-01-09).

[00100] [R08] 3 GPP TS 38.420 v15.2.0 (2019-01-08).

[00101] [R09] 3 GPP TS 38.460 v16.0.0 (2020-01-09). [00102] [R10] 3 GPP TS 38.470 v16.0.0 (2020-01-09).

[00103] [R12] O-RAN Alliance Working Group 1, O-RAN Operations and Maintenance Architecture Specification, version 2.0 (Dec 2019) (“O-RAN- WG1,OAM-Architecture-v02.00”).

[00104] [R13] O-RAN Alliance Working Group 1, O-RAN Operations and Maintenance Interface Specification, version 2.0 (Dec 2019) (“O-RAN- WG1.O1 -Interface-v02.00”).

[00105] [R14] O-RAN Alliance Working Group 2, O-RAN A1 interface: General Aspects and Principles Specification, version 1.0 (Oct 2019) (“ORAN- WG2.A1.GA&P-v01.00”). [00106] [R15] O-RAN Alliance Working Group 3, Near-Real-time RAN

Intelligent Controller Architecture & E2 General Aspects and Principles (“ORAN-WG3.E2GAP.0-V0.1 ”).

[00107] [R16] O-RAN Alliance Working Group 4, O-RAN Fronthaul Management Plane Specification, version 2.0 (July 2019) (“ORAN-WG4.MP.0- v02.00.00”).

[00108] [R17] O-RAN Alliance Working Group (WG) 4, O-RAN Fronthaul Control, User and Synchronization Plane Specification, version 2.0 (July 2019) (“ORAN-WG4.CUS.0-v02.00”).

[00109] [R18] O-RAN WG1, “O-RAN Architecture Description”. [00110] [R19] O-RAN WG2, ““ AI/ML Workflow Description and

Requirements”.

[00111] [R20] O-RAN WG2, “Non-RT RIC Functional Architecture”

TERMINOLOGY

[00112] The term “application” may refer to a complete and deployable package, environment to achieve a certain function in an operational environment. The term “AI/ML application” or the like may be an application that contains some Al/ML models and application-level descriptions.

[00113] The term “machine learning” or “ML” refers to the use of computer systems implementing algorithms and/or statistical models to perform specific task(s) without using explicit instructions, but instead relying on patterns and inferences. ML algorithms build or estimate mathematical model(s) (referred to as “ML models” or the like) based on sample data (referred to as “training data,” “model training information,” or the like) in order to make predictions or decisions without being explicitly programmed to perform such tasks. Generally, an ML algorithm is a computer program that learns from experience with respect to some task and some performance measure, and an ML model may be any object or data structure created after an ML algorithm is trained with one or more training datasets. After training, an ML model may be used to make predictions on new datasets. Although the term “ML algorithm” refers to different concepts than the term “ML model,” these terms as discussed herein may be used interchangeably for the purposes of the present disclosure.

[00114] The term “machine learning model,” “ML model,” or the like may also refer to ML methods and concepts used by an ML- assisted solution. An “ML- assisted solution” is a solution that addresses a specific use case using ML algorithms during operation. ML models include supervised learning (e.g., linear regression, k-nearest neighbor (KNN), descision tree algorithms, support machine vectors, Bayesian algorithm, ensemble algorithms, etc.) unsupervised learning (e.g., K-means clustering, principle component analysis (PCA), etc.), reinforcement learning (e.g., Q-leaming, multi-armed bandit learning, deep RL, etc.), neural networks, and the like. Depending on the implementation a specific ML model could have many sub-models as components and the ML model may train all sub-models together. Separately trained ML models can also be chained together in an ML pipeline during inference. An “ML pipeline” is a set of functionalities, functions, or functional entities specific for an ML-assisted solution; an ML pipeline may include one or several data sources in a data pipeline, a model training pipeline, a model evaluation pipeline, and an actor. The “actor” is an entity that hosts an ML assisted solution using the output of the ML model inference). The term “ML training host” refers to an entity, such as a network function, that hosts the training of the model. The term “ML inference host” refers to an entity, such as a network function, that hosts model during inference mode (which includes both the model execution as well as any online learning if applicable). The ML-host informs the actor about the output of the ML algorithm, and the actor takes a decision for an action (an “action” is performed by an actor as a result of the output of an ML assisted solution). The term “model inference information” refers to information used as an input to the ML model for determining inference(s); the data used to train an ML model and the data used to determine inferences may overlap, however, “training data” and “inference data” refer to different concepts.

[00115] Although an aspect has been described with reference to specific exemplary aspects, it will be evident that various modifications and changes may be made to these aspects without departing from the broader scope of the present disclosure. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. This Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various aspects is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

Claims

CLAIMS What is claimed is:

1. An apparatus for a Non real-time (Non-RT) radio access network intelligence controller (RIC)(Non-RT RIC) in an open radio access network (O- RAN), the apparatus comprising: memory; and, processing circuitry coupled to the memory, the processing circuitry configured to: receive, from a Non-RT RIC application (rApp), an artificial intelligence (AI)/machine learning (ML) training service request, the AI/ML training service request comprising an indication of an AI/ML model structure, training preferences, and a training data description; send a training process identification (ID) to the rApp, the training process ID identifying the AI/ML training service request; and collect training data indicated by the training service request from the radio access network (RAN) via an O1 interface.

2. The apparatus of claim 1 wherein the training data description comprises a data type, a sampling rate, a sampling period, a granularity, and a condition.

3. The apparatus of claim 1 wherein the processing circuitry is further configured to: send an error message in response to the training service request if the training service request cannot be accommodated by the Non-RT RIC, wherein the error message comprises an error code indicating an error with the training service request, and wherein the error comprises: unknown data type, missing elements in training service request, error in training data configuration, or an indication the Non-RT RIC cannot access the training data indicated in the training data description, and wherein the rApp and the Non-RT RIC communicate via an R1 interface.

4. The apparatus of claim 1 wherein the training preferences comprise one or more of the following: an indication of a loss function, an indication of an optimizer, an indication of a number of epochs, an indication of a number of interactions/batches, an indication of a learning rate, an indication of a split ratio of training and validation data, an indication of a gradient clipping, and an indication of a test data set.

5. The apparatus of claim 4 wherein the training preferences further comprise one or more training requirements, the training requirements comprising: an indication of a metric, an indication of training loss requirements, an indication of a validation requirement based on the metric, and an indication of testing requirements based on the metric.

6. The apparatus of claim 5 wherein the processing circuitry is further configured to: train an AI/ML model indicated by the AI/ML model structure in accordance with the training service request; in response to the training being success, send the trained AI/ML model to the rApp; and in response to the training being unsuccessful, send an indication of training failure to the rApp with an error code, the error code indicating an error case.

7. The apparatus of claim 6 wherein the error case is one or more of the following: unknown training process ID, invalid AI/ML model structure, invalid training configuration, failed to satisfy training requirements, and training consume excessive resources or too much time.

8. The apparatus of claim 1 wherein the processing circuitry is further configured to: receive, via an R1 interface, a push AI/ML model to a model repository request from rApp, the push AI/ML model to the model repository request comprising a copy of an AI/ML model; send to the rApp an address/identifier for the AI/ML model if the push of the AI/ML model to the model repository is successful; and send to the rApp an indication of failure if the push of the AI/ML model to the model repository is unsuccessful.

9. The apparatus of claim 8 wherein the processing circuitry is further configured to: receive, via the R1 interface, a pull AI/ML model from the model repository request, the pull AI/ML model from the model repository request comprising the address/identifier of the AI/ML model and an ID of the rApp; send to the rApp the AI/ML model if the pull of the AI/ML model from the model repository is successful; and send to the rApp an error code if the pull of the AI/ML model from the model repository is unsuccessful, wherein the error code indicates invalid address/identifier of the AI/ML model or no access permitted for the rApp with rApp ID.

10. The apparatus of claim 8 wherein the processing circuitry is further configured to: receive, via the R1 interface, a remove AI/ML model from the model repository request, the remove AI/ML model from the model repository request comprising the address/identifier of the AI/ML model and an ID of the rApp; send to the rApp an indication of successful removal if the remove of the AI/ML model from the model repository is successful; and send to the rApp an error code if the remove of the AI/ML model from the model repository is unsuccessful, wherein the error code indicates invalid address/identifier of the AI/ML model or no access permitted for the rApp with rApp ID.

11. The apparatus of claim 1 wherein the processing circuitry is further configured to: receive, via an R1 interface, an AI/ML monitoring request, the AI/ML monitoring request comprising monitoring data description, monitoring metrics and configuration, conditions for AI/ML model retraining and termination, and model management notification destination, wherein the monitoring data description comprises an indication of radio access network (RAN) data; send a model instance ID for model monitoring and management to the rApp if the AI/ML monitoring request is successful; and send an error code to the rApp if the AI/ML monitoring request is unsuccessful, wherein the error code indicates an error case.

12. The apparatus of claim 11, wherein the description comprises an indication of a data type, an indication of a sampling rate, an indication of a period, an indication of a granularity, and an indication of a filtering conditions, and wherein the monitoring metrics and configuration indicates how the Non-RT RIC evaluates the performance of the AI/ML model.

13. The apparatus of claim 11 wherein the processing circuitry is further configured to: collect the RAN data from an RAN via the O1 interface; collect inferred data from the rApp via the R1 interface; and perform AI/ML model performance evaluation based on the RAN data and the inferred data.

14. The apparatus of claim 13 wherein the processing circuitry is further configured to: send a management notification to the rApp via the R1 interface, the management notification of retraining and terminating the AI/ML model performance evaluation and the management notification comprising an error code.

15. The apparatus of claim 1 further comprising transceiver circuitry coupled to the memory; and antennas coupled to the transceiver circuitry.

16. A non-transitory computer-readable storage medium that stores instructions for execution by one or more processors of a non real-time (Non- RT) radio access network intelligence controller (RIC)(Non-RT RIC) in an Open RAN (O-RAN) network, the instructions to configure the one or more processors to perform the following operations: receive, from a Non-RT RIC application (rApp), an artificial intelligence (AI)/machine learning (ML) training service request, the AI/ML training service request comprising an indication of an AI/ML model structure, training preferences, and a training data description; send a training process identification (ID) to the rApp, the training process ID identifying the AI/ML training service request; and collect training data indicated by the training service request from the radio access network (RAN) via an O1 interface.

17. The non-transitory computer-readable storage medium of claim

16 wherein the training data description comprises a data type, a sampling rate, a sampling period, a granularity, and a condition.

18. An apparatus for an Non-time (RT) radio access network intelligence controller (RIC)(Non-RT RIC) application (rApp) in an open radio access network (O-RAN), the apparatus comprising: memory; and, processing circuitry coupled to the memory, the processing circuitry configured to: send, to a Non-RT RIC via an R1 interface, an artificial intelligence (AI)/machine learning (ML) training service request, the AI/ML training service request comprising an indication of an AI/ML model structure, training preferences, and a training data description; and receive a training process identification (ID) from the Non-RT RIC, the training process ID identifying the AI/ML training service request.

19. The apparatus of claim 18 wherein the processing circuitry is further configured to: send, via the R1 interface, a push AI/ML model to a model repository request to the Non-RT RIC, the push AI/ML model to the model repository request comprising a copy of an AI/ML model; receive from the Non-RT RIC an addressZidentifier for the AI/ML model if the push of the AI/ML model to the model repository is successful; receive from the Non-RT RIC an indication of failure if the push of the AI/ML model to the model repository is unsuccessful; send, via the R1 interface, a pull AI/ML model to the model repository request, the pull AI/ML model from the model repository request comprising the address/identifier of the AI/ML model and an ID of the rApp; receive from the model repository the AI/ML model if the pull of the AI/ML model from the model repository is successful; receive from the model repository an error code if the pull of the AI/ML model from the model repository is unsuccessful, wherein the error code indicates invalid address/identifier of the ΑI/ML model or no access permitted for the rApp with rApp ID; send, via the R1 interface, a remove AI/ML model to the model repository request, the remove AI/ML model from the model repository request comprising the address/identifier of the AI/ML model and an ID of the rApp; receive from the model repository an indication of successful removal if the remove of the AI/ML model from the model repository is successful; and receive from the model repository an error code if the remove of the AI/ML model from the model repository is unsuccessful, wherein the error code indicates invalid address/identifier of the AI/ML model or no access permitted for the rApp with rApp ID.

20. The apparatus of claim 18 wherein the processing circuitry is further configured to: send, via an R1 interface, an AI/ML monitoring request, the AI/ML monitoring request comprising monitoring data description, monitoring metrics and configuration, conditions for AI/ML model retraining and termination, and model management notification destination, wherein the monitoring data description comprises an indication of radio access network (RAN) data; receive a model instance ID for model monitoring and management if the AI/ML monitoring request is successful; and receive an error code if the AI/ML monitoring request is unsuccessful, wherein the error code indicates an error case.