EP4728443A1 - Inferencing model selection and management - Google Patents

Abstract

Methods, apparatuses, and systems for inferencing model selection and management are described herein. In one aspect, a method can include sending, by a model selection and management client (MSMC) and to a model selection and management service (MSMS), a management service request; receiving, by the MSMC and from the MSMS, a notification indicating a request for a model selection by the MSMC; and sending, by the MSMC and to the MSMS, a model selection message indicative of a machine learning model selected by the MSMC.

Description

INFERENCING MODEL SELECTION AND MANAGEMENT CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application claims the benefit of U.S. Provisional Application No. 63/508,432 (“Inferencing Model Selection and Management”) filed June 15, 2023, the contents of which is hereby incorporated by reference in its entirety for any and all purposes. BACKGROUND [0002] Machine learning (ML) is a branch of artificial intelligence (AI) aiming at building methods that leverage data to improve performance on some set of tasks. One or more ML algorithms are used to build a model based on sample data, i.e., training data, in order to make predictions or decisions without being explicitly programmed to do so. The process of training a ML model may include data gathering, data cleaning, data preparation, data analysis, model building/training, model evaluation, model deployment, monitoring and update, etc. Figure 1 shows an example ML pipeline that includes the typical steps/services. [0003] After a ML model is trained, it can be deployed to a live environment (i.e., an endpoint) where it can be used for the intended purpose, such as performing an inferencing task. The deployed model has access to the hardware resources it needs as well as the data sources where it can access data for inferencing. The model and inferencing results can be made accessible to the user via an API or via an integrated model with an application/service that is used by the user. [0004] When deploying ML models to perform inferencing tasks, the performance of the model may change over time, resulting in the need to dynamically change or switch the model used for inferencing. This may be caused by the dynamics in the inferencing tasks, such as: data drift, change of data pattern/characteristics/quality; dynamics in the availability/capability of the endpoint environment where the model is deployed; and user changing their requirements/preferences of the inferencing task or performance metrics. [0005] Multiple models could be deployed or made available to the ML user such that a dynamic/adaptive selection can be made to choose the most optimal/suitable model for inferencing under different scenarios or conditions. [0006] In one example, a smart home device is utilizing ML models for recognizing the voice commands of the family members, as shown in Figure 2. Different family member may have different voice features or different speaking habits. To achieve the best recognition accuracy, multiple ML models can be used where each of the models specializes in recognizing the voice commands of a specific family member. For example, when the father is saying a voice command, the device may be configured to use the specialized/personalized model for himself. When there are visitors other than the family members using the device, those models specialized for family member may not work well, then a general voice recognition model could be used instead. [0007] In addition to manually configuring or switching the model to be used, the device could be configured to switch to the most suitable model automatically. For example, with a signal processing function (which could be part of the pre-processing procedure in the inferencing pipeline), the device may be able to identify whether a voice signal comes from a family member and if so, which family member the voice signal comes from. Based on this, the corresponding model specialization can be selected. [0008] In another example, a mobile user is wearing an AR (augmented reality) device that detects objects around the user and provides information of the objects to the user. The detection of objects is done with a ML service that performs real-time inferencing for image recognition. [0009] When the user is moving at a relatively higher speed, the data rate of the inferencing data that is input into the ML model is higher. In order to generate image recognition results in a timely manner, a model with shorter response time may be preferred. Also, moving at a higher speed may indicate that the user is not very interested in the objects around them. Therefore, in this scenario, the user may value the response time of the model more than the accuracy or granularity of results. A light-weight model (e.g., a model with fewer layers and neurons) could be selected that guarantees short response time at the cost of inference accuracy or granularity. [0010] When the user is moving at a relatively lower speed, the inferencing data rate may not be very high and a model with longer processing time may be acceptable. Moving at a lower speed may indicate that the user is interested in the objects around them. In this case, the user may prefer accuracy over response time and may request results with finer granularity. For example, the user stops at a crowd of penguins when touring an aquarium. In addition to recognizing the penguins, the user may want to know the exact species of the penguins (e.g., emperor penguin, Adelie penguin, etc.), which may require a more complicated model (e.g., a model with more layers and neurons) at the cost of longer response time. [0011] In this real-time scenario, it is preferred that the switching of models can be done automatically based on the detection of a user’s speed and/or the inferencing data rate. For example, when the user’s speed or data rate is high, a base model for animal recognition will be used to provide recognition results to the AR service as soon as possible, while the result may only provide a rough categorization (e.g., “penguin”, “whale”). When the user’s speed or data rate is low, the recognition result of the baseline model can be further fed into a fine-tuned layer to generate results with finer granularity (e.g., “emperor penguin”, “orca”), although it may take a longer time to generate the final results. The example is shown in Figure 3. [0012] Existing ML services may support the update of models when the inferencing data changes in patterns or characteristics. When data drift is detected, the model may be updated or re-trained to fit better the new data. However, the drift is defined with respect to a certain baseline dataset (usually the training dataset of the model). In the scenarios such as Use case 1, there could be different data patterns where each requires a specialized model, and there is no single baseline dataset. In this case, dynamically switching between models would be more suitable than replacing the model with an updated one. [0013] Existing ML services may also support the deployment of multiple models, such as deploying multiple model variants for testing and comparing their performance or for performing different inferencing tasks. A user may be able to select which model should be used and/or specify the traffic distribution among different models. However, there is a lack of support to enable a service to provide an automated and adaptive selection of models based on the real-time conditions such as data, endpoint status, and the user’s requirements. Currently, the selection of models relies on manual configuration by the user and there is a lack of mechanisms to enable automated switching and management of models. SUMMARY [0014] This disclosure defines an adaptive ML model selection and management service that provides assistance in selecting and managing multiple ML models for deployment and inferencing. The service is capable of: deploying one or more ML models where each is trained with a different dataset (specialization) or a different hyperparameter configuration (e.g., using transfer learning); monitoring and collecting information such as inferencing data, the endpoint, and requirements/preferences of the inferencing tasks, to assist the decision making of model selection or management; supporting the dynamic selection/switching of models used for inferencing tasks; managing the deployed models, providing trigger information for deploying a new model, updating one or more deployed models, or removing a model from the deployment; and automatically performing model selection and management operations on behalf of the requestor based on the requestor’s configurations and policies. [0015] In some cases, a model selection and management service (MSMS) may be defined to perform: receiving a model selection and management request from a requestor. The request may include information of an inferencing task that is requested by a ML user, such as the type/category of inferencing task, information of inferencing data, requirements of the inferencing task, etc. The request may include information of the ML model(s) and/or inferencing pipeline(s) that may be deployed to perform the inferencing task, information of the endpoint(s) where the model may be deployed. The request may include notification policies, which may specify when to send a notification for model selection and management operations, which entity the notification should be sent to, and what information should be provided in the notification. The requestor may be the ML user, or an entity associated with the endpoint, or an entity associated with the ML service, etc. [0016] The MSMS may be defined to perform: deploying models to perform inferencing tasks, monitoring the inferencing pipeline and collecting information based on the requirements in the notification policies. The information may include characteristics or quality measurements of the inferencing data and/or results, model performance, endpoint status, inferencing data and/or the corresponding ground truth, etc. [0017] The MSMS may be defined to perform: sending a notification requesting model selection. The notification may include information that is collected by the MSMS. Based on the notification policies, the notification may be sent according to a pre-defined schedule, periodically, when new inferencing data is available, when model is deployed, when data drift is detected, when inferencing data rate changes, when the confidence level of the inferencing result is low, when change of endpoint status/conditions is detected, when change of inferencing requirements is detected, etc. [0018] The MSMS may be defined to perform: receiving a message indicating the model selection decision, applying the selected model to the inferencing task. The decision may include an indicator of whether inferencing data/results and the corresponding ground truth need to be collected. [0019] The MSMS may be defined to perform: sending a notification requesting management operations. The management operations may include deploying a new model, updating a deployed model, or removing a model from the deployment, adding/removing an endpoint. The notification may include information that is collected by the MSMS. The notification may include inferencing data/results and the corresponding ground truth that are collected by the MSMS. Based on the notification policies, the notification may be sent according to a pre-defined schedule, periodically, when data drift is detected, when the confidence level of the inferencing result is low, when change of endpoint status/conditions is detected, etc. [0020] The MSMS may be defined to perform: receiving an updated model selection and management request. The request may include updated model/pipeline/endpoint information. [0021] In another aspect, a MSMS may be defined to perform: receiving a model selection and management request from a requestor. The request may include information of an inferencing task that is requested by a ML user, such as the type/category of inferencing task, information of inferencing data, requirements of the inferencing task, etc. The request may include information of the ML model(s) and/or inferencing pipeline(s) that may be deployed to perform the inferencing task, information of the endpoint(s) where the model may be deployed. The request may include selection policies, which may specify when the MSMS should perform model selection and the corresponding selection criteria. The request may include management policies, which may specify the conditions of triggering management operations. The requestor may be the ML user, or an entity associated with the endpoint, or an entity associated with the ML service, etc. [0022] The MSMS may be defined to perform: deploying models to perform inferencing tasks, monitoring the inferencing pipeline and collecting information that will be used for performing model selection and management operation, which is indicated in the selection policies and management policies. The information may include characteristics or quality measurements of the inferencing data and/or results, model performance, endpoint status, inferencing data and/or the corresponding ground truth, etc. [0023] The MSMS may be defined to perform: determining model selection should be performed and selecting the model(s) based on collected information and the selection policies. [0024] The MSMS may be defined to perform: determining management operations should be performed and performing the management operations based on collected information and the management policies. The management operations may include deploying a new model, updating a deployed model, or removing a model from the deployment, adding/removing an endpoint. [0025] The MSMS may be defined to perform: sending a notification regarding the performed model selection and/or management operations, [0026] The MSMS may be defined to perform: sending a message requesting for new/updated model or endpoint. The request may be sent to the requestor or the ML user. The request may be sent to a model training service. The request may further include collected inferencing data and the corresponding ground truth. BRIEF DESCRIPTION OF THE DRAWINGS [0027] Figure 1 depicts an example Machine Learning (ML) pipeline; [0028] Figure 2 depicts a ML-based voice recognition for family members; [0029] Figure 3 depicts a ML-based animal image recognition; [0030] Figure 4 depicts adaptive model selection and management service entities and interactions; [0031] Figures 5-10 depict processes for inferencing model selection and management; [0032] Figures 11-14 depict graphical user interfaces (GUIs) for inferencing model selection and management; [0033] Figure 15A depicts an example communications system in which the methods and apparatuses described and claimed herein may be an aspect of; [0034] Figure 15B depicts a block diagram of an example apparatus or device configured for wireless communications; [0035] Figure 15C depicts a system diagram of an example radio access network (RAN) and core network; [0036] Figure 15D depicts a system diagram of another example RAN and core network; [0037] Figure 15E depicts a system diagram of another example RAN and core network; [0038] Figure 15F depicts a block diagram of an example computing system; and [0039] Figure 15G depicts a block diagram of another example communications system. DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS [0040] The present disclosure provides an adaptive inferencing model selection and management service (MSMS), which provides assistance to a ML user or ML deployment service in the scenario where multiple models are available or needed for an inferencing task. Particularly, the proposed service may assist in the process of determining which model(s) should be deployed and which model(s) should be used for an inferencing task based on dynamic information such as the user’s requirements, inferencing data characteristics, and/or endpoint conditions. Moreover, the proposed service may support the management process of the models (e.g., deploying a new model, updating a model, removing a model from deployment), such as detecting trigger conditions for management operations, monitoring and/or collecting inferencing data for creating or re-training a model, performing a management operation on behalf of the user, etc. [0041] The proposed functionality can be implemented as a service. The service may be integrated in another ML service (or a specific deployment service, endpoint management service, life cycle management service of ML) or be deployed as a standalone service that may interact with other services that are relevant to the deployment and management of ML models. The service may be integrated in or supporting an analytics service in an application and service enablement layer. The service may be hosted in the cloud, on the edge of the network, on gateways or end devices. Figure 4 shows the main entities involved in the proposed service and their interactions. [0042] Requestor: The requestor initiates the request for model selection and management service. The requestor could be the ML user who initiates the request for inferencing tasks (and receives inferencing results). The ML user could be a human user or an application/service that may invoke the ML service or the deployed models. The requestor could also be a service/entity related to model deployment, such as a deployment service, an endpoint management service, etc. [0043] Endpoint: An endpoint is an entity that provides an interface for a ML user to use the trained ML models for inferencing tasks. The user may send inferencing requests (which optionally contains inferencing data) to the endpoint and receive inferencing results from the endpoint (or instruct the endpoint to send the inferencing results to a specific destination). The hosting environment of an endpoint provides storage, computation, communication resources to the endpoint that are necessary to support the inferencing task. An endpoint can be hosted at a cloud-based service, a web service, edge server, gateway, user’s device, etc. [0044] Inferencing Pipeline: ML models can be deployed to an endpoint or multiple endpoints (e.g., for split learning) to perform the inferencing task. When deploying a model, the corresponding pre/post-processing functions can be deployed together with the model, as an inferencing pipeline. An inferencing pipeline takes the inferencing data as input, and generates inferencing results as output. [0045] In the case where multiple models are available to be used for an inferencing task, they can be deployed jointly in a single inferencing pipeline. The models may share some or all of the pre/post-processing logic (as shown in Figure 4), or each model may have its specific pre/post-processing functions (not shown in figure). Alternatively, the models may be deployed as individual pipelines. [0046] Model Selection and Management Service (MSMS): The MSMS is the entity that provides the model selection and management service proposed in this disclosure. The MSMS interacts with Model Selection and Management Clients (MSMCs). The MSMS may receive requests from the MSMCs of the requestor and interact with endpoint MSMCs (directly or via a deployment or endpoint management service) to provide functionalities such as: [0047] Model selection: When multiple models are available for deployment to perform an inferencing task, the MSMS may assist with the process of selecting one or more models to deploy to the endpoint. [0048] When multiple models have been deployed to the endpoint, the MSMS may assist with the process of selecting one or more models from the deployed models to perform the inferencing task. [0049] Model management: The MSMS may assist with the management process of adding a new model to deployment, updating a deployed model, or removing a model from deployment. [0050] Additionally, the MSMS may interact with MSMCs of other services or entities that are relevant, such as the model deployment service, model registry, endpoint (or the corresponding hosting environment) management service, model training service, monitoring service, etc. The requestor could also be one of these services/entities. [0051] Model Selection and Management Client (MSMC): The MSMC is the entity associated with the requestor, the endpoint or other relevant services which the MSMS may interact with to enable the service. Information Elements [0052] This section introduces the information elements that may apply to the procedures described in the following sections. The defined information elements may be applied in the messages that are exchanged between different entities in the system. Endpoint profile [0053] The MSMS may use endpoint profiles to track information of the endpoint and use this information when managing model selection and deployment. An endpoint profile describes information associated with an endpoint and its corresponding hosting environment. Endpoint profiles can be provided by the requestor MSMC to the MSMS and can be maintained by the MSMS to record information of the endpoint and enable inferencing model selection and management. The information elements for an endpoint profile are shown in Table 1. Table 1. Endpoint Profile Information Element Parameter Description End Point Identifier Identifier of the endpoint, may also include the IP address/port number and/or URI of the endpoint. Supported Models A list of identifiers (and/or the corresponding model profiles) of the ML models that can be deployed to this endpoint. Deployed Models A list of identifiers (and/or the corresponding model profiles) of the ML models that are currently deployed to this endpoint. Deployed Inferencing A list of identifiers (and/or the corresponding profiles) of the Pipelines inferencing pipelines that are currently deployed to this endpoint. This IE may also indicate the information of Deployed Models. Hosting Environment Information of the hosting environment of the endpoint, which provides storage, computation, communication, operating environments (e.g., operating system, system packages), and other resources that are necessary for the endpoint to support the deployed inferencing pipelines / models and run inferencing tasks, such as a cloud server, an edge server, a gateway, a device, etc. The information may include an identifier, address, description of the Hosting Environment, etc. Associated entity Information (e.g., identifier, address) of the entity associated with the endpoint with which the MSMS may interact, such as the management service associated with the endpoint or its Hosting Environment. Capability Information Descriptions of the capability of the endpoint and its hosting environment in supporting the model deployment and inferencing tasks, such as the computation resources, storage capacity, communication bandwidth, etc. Availability Information Descriptions of the availability or schedule of the endpoint and its hosting environment in supporting the model deployment and inferencing task. For example, all the resources described in the Capability Information are available for inferencing tasks during weekdays but only 50% of the resources are available during weekends. Context Information Context information of the endpoint and its hosting environment which is not reflected in the Capability/Availability Information but may be used by the MSMS in supporting the model deployment and inferencing tasks, such as the geographical location, operational cost, stability, affiliation information, etc. Status Information Information of the status or condition or the endpoint and/or its Hosting Environment, such as the workload, resource utilization, communication quality, energy consumption, etc. Inferencing pipeline profile [0054] The MSMS may use inferencing pipeline profiles to track information of the inferencing pipeline, manage the deployment of the pipeline, and manage data associated with the inferencing task. Inferencing pipeline profiles can be provided by the requestor MSMC to the MSMS, or generated by the MSMS when deploying models. Inferencing pipeline profiles can be maintained by the MSMS to record information of the pipeline and enable inferencing model selection and management. The information elements for an inferencing pipeline profile are shown in Table 2. Table 2. Inferencing Pipeline Profile Information Element Parameter Description Inferencing Pipeline Identifier of the inferencing pipeline, may also include the IP Identifier address/port number and/or URI of the pipeline. Pipeline Description Description of the inferencing pipeline. Requestor/User Identifier Identifiers of the requestor and/or user associated with this pipeline, who requires the deployment of this pipeline and uses this pipeline for inferencing tasks. Pipeline Requirements Describes requirements and/or preferences for deploying the pipeline or using the pipeline for inferencing tasks, such as storage requirement, computation power requirement, requirements on the inferencing data (e.g., data format, data rate, data characteristic), etc. The requirements can be specified as a range, and may include minimum/maximum requirements or preferred setting. The pipeline requirements can be generated based on aggregating/combining the corresponding Model Requirements of the models in this pipeline. Pipeline Control/Status Control and/or indicator of the current status of the inferencing pipeline, such as: ^ Available and ready to be deployed ^ Deployed and ready for inferencing task ^ Selected for performing inferencing task ^ Active/ongoing inferencing task Pipeline Components Descriptions of the components of the pipeline, including the model(s) and the pre/post-processing services/functions in the pipeline. May also include descriptions of the structure of the pipeline or how the models and processing functions in the pipeline are connected, such as linear, tree-structure, hierarchical, etc. Selected Models Identifier(s) of the one or more models in this pipeline that are selected for performing inferencing task. Inferencing Data Information of the inferencing data associated with this pipeline, Information including the characteristics and quality measurements of the data. May also include an indicator of whether the inferencing data is being collected/logged. If the inferencing data is being collected, may specify the location of where the data is stored. Inferencing Result Information of the inferencing results associated with this pipeline, Information including the characteristics and performance measurement of the results. May also include an indicator of whether the inferencing results are being collected/logged. If the inferencing results are being collected, may specify the location of where the results are stored. Ground Truth Information Information of the ground truth of the inferencing task associated with this pipeline, including the location (or identifier of a service/entity) from where the ground truth can be obtained. May also include an indicator of whether the ground truth is being collected. If the ground truth is being collected, may specify the location of where it is stored. Inferencing Performance Information of the pipeline’s inferencing performance, such as Information confidence level (evaluated without ground truth), precision related metrics (evaluated with inferencing data and ground truth), response time, etc. The Inferencing Performance Information can be generated based on the performance of the selected model(s) that performs the inferencing tasks in the pipeline. Model Profile [0055] The MSMS may use model profiles to track information of the available and deployed ML models for an inferencing task and manage the models. Model profiles can be provided by the requestor MSMC to the MSMS, and maintained by the MSMS to record information of the models and enable inferencing model selection and management. The information elements for a model profile are shown in Table 3. Table 3. Model Profile Information Element Parameter Description Model Identifier Identifier of the model. Model Description Description of the model, such as the type of the model, what inferencing tasks the model can be used for, etc. Model Information Information such as the algorithm of the model, structure/size of model, parameters of the model, specialization, etc. Model Location Specifies an entity or location where the model can be fetched/retrieved from, such as a model registry service or a model repository. Pipeline Information If the model is (to be) deployed in an inferencing pipeline, the information of the pipeline may be specified, such as identifier of the pipeline or information (identifiers) of the pre/post-processing functions/services. Model Requirements Describes requirements and/or preferences for deploying the model or using the model for inferencing tasks, such as storage requirement, computation power requirement, requirements on the inferencing data (e.g., data format, data rate, data characteristic), etc. The requirements can be specified as a range, and may include minimum/maximum requirements or preferred setting. Model Control/Status Control and/or indicator of the current status of the model, such as: ^ Not trained, collecting data (building training dataset) ^ Available and ready to be deployed ^ Deployed and ready for inferencing task ^ Selected for performing inferencing task ^ Active/ongoing inferencing task Training Data Information of the training/baseline dataset associated with this model, Information such as the format, characteristics of the data. May also include the storage location of the training dataset. Training Performance Information of the model’s training performance, such as precision related Information metrics (evaluated with the training/baseline dataset), response time, etc. May also include the storage location of this information. Inferencing Information of the model’s inferencing performance, such as confidence Performance level (evaluated without ground truth), precision related metrics Information (evaluated with inferencing data and ground truth), response time, etc. May also include the storage location of this information. Management service request [0056] The requestor MSMC may send a request to the MSMS for supporting an inferencing task. The request may specify the following information, as shown in Table 4. Table 4. Management service request Information Element Parameter Description Requestor Identifier Identifier of the originator of the request. For example, identifier of a user, endpoint, or deployment service originating the request. Alternatively, the identifier may be an identifier of a MSMC corresponding to the user, endpoint, or deployment service originating the request. Requestor Description Description of the requestor. For example, which type of entity originates the request (e.g., the user, the endpoint, the deployment service, etc.). Requestor Information Identifier of the requestor, which can be used to identify a follow-up or updated request. May include additional information (e.g., context information) of the requestor that may be used by the MSMS. For example, an Application ID if the requestor is an application. User Information Identifier and other information (e.g., context information) of the user associated with this request that may be used by the MSMS in model selection and management. For example, an Application ID if the user is an application. Inferencing task Identifier and other information of the inferencing task associated with Information this request that is known to the Requestor and may be used by the service in model selection and management, such as the type of the inferencing task (e.g., classification, regression) or the category of the task (e.g., image/voice recognition, numerical value prediction), etc. Inferencing Data Information of the inferencing data associated with this request or Information inferencing task that is known to the requestor, such as the format of data (e.g., image, numerical values, etc.), or the category (e.g., medical image data, animal image data, etc.). Inferencing Requirements Specifies the requirements on the inferencing task, such as minimum confidence level of the result (i.e., inferencing precision), maximum response time (i.e., inferencing latency), etc. The information can be obtained from the user by the requestor if the requestor is not the user. The requestor may specify conditional or dynamic requirements which may indicate more than one model is needed for deployment. For example, the requirements may indicate that a lower confidence level is required when the inferencing data rate is high, and a higher confidence level is required when the inferencing data rate is low. In another example, the requestor may specify higher tolerance of false alarm rate for data deviating from the training/baseline dataset, and a lower tolerance for data that is similar to the baseline/training dataset. Inferencing Data Indicates whether/when the inferencing data, inferencing results and the Collection Requirements corresponding ground truth should be collected. May also include information and configurations related to the data collection, such as the location of the data to be collected (data sources), data collection rate, filter information, etc. May also specify whether the collected data is to be used to update/re-train a model or to create/train a new model. The requestor may request the service to start or stop collecting inferencing data/result and the corresponding ground truth by sending updated management request to the service and configuring this IE in the updated request. Endpoint Information A list of identifiers/profiles of the endpoints where the models are deployed or to be deployed. Pipeline Information A list of identifiers/profiles of the inferencing pipelines that are deployed or to be deployed. Model Information A list of identifiers/profiles of the models that are deployed or to be deployed. The requestor may specify an entity or location where the MSMS can fetch/retrieve models from, such as a model registry service or a model repository. Model Selection Decision Information of the (initial) model selection decision that specifies which models should be deployed to which endpoint and which model(s) should be used for inferencing. If this information is not provided in the initial request, the MSMS may determine the initial selection for model deployment. Model selection decision may include the selection of a human review process (augmented AI) to replace or complement the model-based inferencing process/results. Notification Policy Specifying how the MSMS may send a notification or notify the decision entity to make a decision on model selection or management operation, which may include the conditions when a notification should be sent, who is the notification target (in addition to the decision entity), what decision is to be made by the decision entity (or what actions are expected from the decision entity), and what information should be included in the notification. ^ Example conditions: explicit request from the requestor, pre- defined schedule, periodical, when new inferencing data is available (inferencing data input to pipeline detected), model deployed, model switched, data drift detected, change of inferencing data rate, low confidence level, change of endpoint status/conditions detected, change of inferencing requirements, etc. ^ Example notification target: requestor, user, deployment service, endpoint management service, etc. The request may specify the identifier/address of the notification target. A notification may be sent to more than one targets. Different notification target or decision entity may be specified for different conditions. ^ Examples of expected action: select model for deployment, select model for inferencing task, initiate a management operation, create a new model, deploy a new model, update a model, remove a model from deployment, add a new endpoint, etc. ^ Examples of information to be included in the notification: characteristics or quality measurement of inferencing data, inferencing performance, endpoint status, collected inferencing data and ground truth, etc. The conditions, expected action, and information to be included can be specified separately or in tuples. Selection Policy Defines policies based on which the service may make a selection of models, including the conditions of when the service should make a selection, what action is expected from the service when making the selection, and the selection criteria. Example conditions: explicit request from the requestor, pre-defined schedule, periodical, when new inferencing data is available (inferencing data input to pipeline detected), data drift detected, change of inferencing data rate, low confidence level, change of endpoint status/conditions detected, change of inferencing requirements, no response from decision entity after sending notification, etc. Examples of expected action: select model for deployment, select model for inferencing task, collect inferencing data/results, collect ground truth, request human review or ground truth, etc. Examples of selection criteria: select the model whose training/baseline data is the most similar to the inferencing data, select the model with minimum response time, select model with highest confidence level, select the specialization branch based on intermediate result, etc. The conditions, expected action, and selection criteria can be specified separately or in tuples. Management Policy Defines policies based on which the service may trigger or perform a management operation on the models, including the conditions of when the service should trigger or perform a management operation and what action is expected from the service. Example conditions: explicit request from the requestor, pre-defined schedule, periodical, when new inferencing data is available (inferencing data input to pipeline detected), data drift detected, change of inferencing data rate, low confidence level, change of endpoint status/conditions detected, change of inferencing requirements, no response from decision entity after sending notification, high/low model usage, high/low endpoint workload, inferencing performance drift, etc. Examples of expected action: trigger creation of a new model, request for one or more new models to be deployed, deploy a new model, trigger model update/re-training, remove a model from deployment, request for new endpoints, etc. The conditions and expected action can be specified separately or in tuples. Authorization Token Providing an authorization token to the MSMS so that it will be authorized to manage the models and obtain information from the other relevant services/entities. Model Selection and Management Procedure [0057] Depending on the decision entity of model selection and management operations, the procedure can be decision-driven (decision made by a decision entity such as the requestor, the user or endpoint), policy-driven (decision made by the MSMS), or a combination of these two. [0058] The main steps of the decision-driven management procedure are shown in Figure 5. In this procedure, the decisions of model selection (for deployment and inferencing) and management operations are made by a decision entity (e.g., the requestor, the user, the endpoint, etc.). The MSMS may monitor trigger conditions for notifying the decision entity to make a decision and collect and provide necessary information to the decision entity for making the decisions. Note that the steps in Figure 5 may be performed repeatedly or in an order different from what is shown. [0059] Step 1: The requestor (e.g., via its corresponding MSMC) may send a request to the MSMS. In the request, the requestor may include information of the models and inferencing pipeline that are to be deployed and managed (model profiles, inferencing pipeline profile), information of the endpoint(s) (e.g., endpoint profile(s)), and an initial selection decision specifying which model(s) should be deployed to which endpoint(s) and which model(s) should be used for inferencing. The requestor may define a Notification Policy in the request, where the policy specifies when the MSMS should notify the decision entity to make a decision on model selection and/or management operation, and what information should be included in the notification message. The request may also include information as described in Table 4. [0060] Step 2: The MSMS may deploy models and/or inferencing pipelines to the endpoint(s) via the MSMC(s) associated with the endpoint(s) according to the initial selection specified by the requestor, and may start monitoring the inferencing pipeline and endpoint(s) to collect information as indicated in the Notification Policy. For example, if the Notification Policy specifies a notification condition to be the detection of endpoint status change, then the MSMS may start monitoring the endpoint and/or collecting status information of the endpoint. In another example, if the Notification Policy specifies a notification condition to be the detection of data drift, then the MSMS may start monitoring data quality and evaluating the inferencing data/results quality for data drift. [0061] The MSMS may send a notification informing the user (directly or via the requestor) that the models/pipelines have been deployed so that the user may start using the inferencing pipeline and sending inferencing requests to the endpoint. [0062] Step 3: When conditions for model selection defined in the Notification Policy are met, the MSMS may send a notification to the MSMC associated with the decision entity and may request the decision entity to select a model for inferencing. In the notification message, the MSMS may include the required information as indicated in the Notification Policy, such as inferencing data/results quality measurements, endpoint status/conditions, etc. [0063] Step 4: The decision entity may send the model selection decision to the MSMS. The decision may be sent as an updated management service request with updated Model Selection Decision and/or updated model/pipeline information. Other updated information may also be included in this message. For example, the Inferencing Data Collection Requirements may be updated to ask the MSMS to start/stop collecting inferencing data/results and the corresponding ground truth. [0064] Step 5: The MSMS may update the inferencing pipeline profile and model profile based on the model selection decision. The MSMS may also start/stop collecting inferencing data/results as requested by the decision entity in step 4. [0065] Step 6: When conditions for management operations defined in the Notification Policy are met, the MSMS may send a notification to the corresponding decision entity (or the associated MSMC) and may request the decision entity to determine/perform a management operation. In the notification message, the MSMS may include the required information as indicated in the Notification Policy, such as model usage, inferencing performance, endpoint workload, etc. [0066] Step 7: The decision entity may determine and perform one or more management operations, such as sending an updated management request with updated Model/Pipeline/Endpoint Information to the MSMS. [0067] Step 8: The MSMS may update the profiles and deployment based on the updated request. [0068] In addition to the entities shown in Figure 5, the MSMS may coordinate with the MSMC(s) associated with a model registry, deployment service, endpoint management service, and/or other relevant services/entities to perform the above steps. [0069] The main steps of the policy-driven management procedure are shown in Figure 6. In this procedure, the MSMS may detect trigger conditions and make decisions of model selection (for deployment and inferencing) and may perform management operations in an autonomous manner (e.g., on behalf of the requestor based on pre-defined policies and the information collected by the MSMS. Note that the steps in Figure 6 may be performed repeatedly or in an order different from what is shown. [0070] Step 1: The requestor (via its corresponding MSMC) may send a request to the MSMS. The requestor may define a Selection/Management Policy in the request, where the policy specifies when the MSMS should make a decision on model selection and/or management operation, and how to make the decision. The requestor may not include an initial Model Selection Decision in the request. This absence may indicate that the requestor is requesting the management service to determine the initial selection, i.e., which model(s) should be deployed to which endpoint(s) and which model(s) should be used for inferencing. The request may also include information as described in Table 4. [0071] Step 2: Based on the Selection Policy and/or other information in the request, the MSMS may determine the initial selection of models for deployment. For example, the MSMS may choose one or more models based on the Inferencing Requirements of the user, information of the user and/or requestor, information of the inferencing task or the endpoint, or by comparing Model Requirements in the Model Profile with Capability Information in the Endpoint Profile, etc. The MSMS may send a notification to the user indicating which models have been selected for deployment. [0072] Step 3: The MSMS may deploy the selected models and/or inferencing pipelines to the endpoint(s) via the MSMC associated with the endpoint(s) according to the initial selection determined in step 2, and may start monitoring the inferencing pipeline and endpoint to collect information as indicated in the Selection/Management Policy. For example, if the Selection/Management Policy specifies that the model selection or management decision should be made based on the response time of the deployed models, then the MSMS may start monitoring the performance of the model/pipeline in terms of response time. In another example, if the Selection/Management Policy specifies that the model selection decision should be made based on comparing the characteristics of the inferencing data with the training/baseline datasets of the deployed models, then the MSMS may start monitoring and collecting information of the inferencing data. [0073] Step 4: When the conditions for model selection defined in the Selection Policy are met, the MSMS may choose one or more models to perform the inferencing task based on the Selection Policy and information collected by the MSMS. [0074] For example, the MSMS may compare the inferencing data with the training/baseline dataset of each of the deployed models, evaluate the similarities, and select the model whose training/baseline dataset is the most similar to the inferencing data. In another example, the MSMS may retrieve information from the pre-processing functions to determine which model specialization should be selected. In yet another example, the MSMS may obtain inferencing data rate and workload information from the endpoint, filter out the models that are not able to generate inferencing results in time, and select the most accurate model among the remaining ones. [0075] Step 5: The MSMS may notify the requestor (via its corresponding MSMC) of the selected model for inferencing. [0076] Step 6: When the conditions for management operations defined in the Management Policy are met, the MSMS may determine, trigger and optionally perform management operation(s) based on the Management Policy and information collected by the MSMS. [0077] For example, the MSMS may collect inferencing data and the corresponding ground truth for the data instances where the models generate a low-confidence or incorrect inferencing results. When the amount of the collected data reaches a pre-defined value (defined in Management Policy), the MSMS may trigger the creation of a new model or update/re-training of the existing model(s). [0078] In another example, the MSMS may detect that a deployed model has not been selected/used for a long period of time, and may remove the model from deployment. [0079] In yet another example, the MSMS may detect that the endpoint where the model(s) are deployed is overloaded which impacts the performance of the inferencing tasks (e.g., longer response time). The MSMS may determine to request for new/additional endpoints to offload the inferencing tasks. [0080] Step 7: The MSMS may notify the requestor (via the corresponding MSMC) of the determined/performed management operation(s). If the determined management operation requires new/updated model(s) to be deployed or requires additional endpoint for deployment, the MSMS may indicate in the notification a request for new/updated model(s) or endpoint(s). In the notification, the MSMS may include collected inferencing data/results and the corresponding ground truth which can be used by the requestor to create/update the model. Alternatively, the MSMS may send the request for new/updated model to a training service. [0081] Step 8: The requestor may send an updated management service request to the MSMS with updated Model Information and/or Endpoint Information as requested by the MSMS. [0082] Step 9: The MSMS may update the profiles and deployment based on the updated request. [0083] In addition to the entities shown in Figure 6, the MSMS may coordinate with the MSMC(s) associated with a model registry, deployment service, endpoint management service, and/or other relevant services/entities to perform the above steps. Adaptive Model Selection and Management Features [0084] This section defines several features that may be supported by the proposed MSMS. The features may be combined with one another to support more advanced capabilities. [0085] When multiple models are deployed and each model specializes in a certain subset of data, the MSMS may provide assistance in selecting the most suitable model to perform the inferencing tasks. Figure 7 shows an example procedure. Note that the steps may occur in a different order than the one shown in the figure. [0086] Step 0: The requestor (via its corresponding MSMC) may configure a management service request with the MSMS and provide the information found in Table 4. The MSMS may also configure the endpoint according to the requirements of the request. [0087] Step 1: Inferencing data is received by the endpoint and fed into the inferencing pipeline deployed on the endpoint. [0088] Step 2: Data processing or measurements may be performed by the data functions in the inferencing pipeline or other relevant services. The MSMS may request data processing or measurement results from the MSMC associated with the endpoint or the relevant services to extract information of the inferencing data. Alternatively, the MSMS may retrieve inferencing data from the endpoint (via the corresponding MSMC) and perform processing/measurement (by itself or utilizing other services) on the data. [0089] Step 3: The MSMS may compare the information of the inferencing data with the Training Data Information in each of the deployed model’s profile to determine the most suitable model. The decision can be made by the MSMS based on the Selection Policy, or by the decision entity. For example, the Selection Policy may define that the decision is based on evaluating the similarity (similarity score) between the inferencing data and the training/baseline data of the deployed models. The model with the highest similarity score may be selected. [0090] Step 4: If the decision of model selection is to be made by the decision entity, the MSMS may send a message to the decision entity (via its corresponding MSMC) requesting for model selection. In the request, the MSMS may include the information of the inferencing data such as the information obtained in steps 2 and 3. [0091] Step 5: The decision entity may respond with a model selection decision to the MSMS. [0092] Step 6: The MSMS may instruct the endpoint to perform inferencing task with the selected model. [0093] Steps 1 to 6 may be repeated for each inferencing data instance or each batch of inferencing data, or according to a pre-defined schedule or trigger, depending on the configurations in Notification Policy and/or Selection Policy. [0094] In some cases, more than one model may be selected to perform the inferencing task, e.g., to increase the confidence level or accuracy of the inferencing result. For example, the Selection Policy may define that all models with a similarity score higher than a pre-defined value will be selected, and the inferencing result may be generated as a weighted sum of the inferencing results from the selected models, where the weights are defined as proportional to the similarity scores. In this case, the MSMS may instruct the endpoint to generate the weighted sum as the inferencing result. [0095] An inferencing pipeline may consist of multiple models that are connected sequentially or in a tree/hierarchical structure. The decision of which model branch should be taken can be made based on the intermediate result from some of the models. The MSMS may collect the intermediate result and provide it to the decision entity or make a decision based on the Selection Policy. [0096] In addition to the characteristics of the inferencing data, other factors may also affect the selection of the model(s), such as the endpoint conditions or the user’s requirements, as different model options may vary in their resource requirement/usage or performance. The MSMS may provide assistance in selecting the most suitable model to perform the inferencing task. Figure 8 shows an example procedure. Note that the steps may occur in a different order than the one shown in the figure. [0097] Step 0: The requestor (via its corresponding MSMC) may configure a management service request with the MSMS and provide the information found in Table 4. The MSMS may also configure the endpoint according to the requirements of the request. [0098] Step 1: Inferencing data is fed into the inferencing pipeline deployed on the endpoint. [0099] Step 2: The MSMS may monitor the status/conditions of the endpoint and the inferencing pipeline, and collect information such as the inferencing data rate, processing time of inferencing task, resource usage, workload, etc. The MSMS may directly monitor the endpoint or request information from other services that are monitoring/managing the endpoint or its hosting environment via the corresponding MSMC. [00100] Step 3: The MSMS examines the Inferencing Requirements and/or the model requirements in each of the model’s profile to determine the most suitable model. The decision can be made by the MSMS based on the Selection Policy, or by the decision entity. For example, the Selection Policy may specify that the response time from receiving inferencing data instance to generating inferencing result should not exceed a pre-defined time limit. The MSMS may filter the models with this requirement. If more than one model could satisfy the requirements, the one with the highest precision could be selected. [00101] Step 4: If the decision of model selection is to be made by the decision entity, the MSMS may send a message to the decision entity (or the corresponding MSMC) requesting for model selection. In the request, the MSMS may include the information obtained in steps 2 and 3. [00102] Step 5: The MSMS receives model selection decision from the decision entity. [00103] Step 6: The MSMS instructs the endpoint (via the corresponding MSMC) to perform inferencing task with the selected model. [00104] Steps 1 to 6 may be repeated for each inferencing data instance or each batch of inferencing data, or according to a pre-defined schedule or trigger, depending on the configurations in Notification Policy and/or Selection Policy. Dynamic model selection for deployment [00105] A ML user may encounter various inferencing data that requires different models for the inferencing tasks. The user or the corresponding requestor may request to deploy all the models that are potentially needed to the endpoint and leverage the MSMS for dynamic selection/switching among the models. In some scenarios, the user or requestor may want to limit the number of models that are deployed, e.g., due to limited storage capacity of the endpoint (or its Hosting Environment). In this case, the MSMS may provide assistance in dynamically selecting model(s) for deployment in real-time and removing unused model from deployment to release storage space. Figure 9 shows an example procedure. Note that the steps may occur in a different order than the ones shown in the figure. [00106] Step 1: The requestor (via it’s corresponding MSMC) may send a request to the MSMS. The request may include information of multiple models that are potentially needed, and which are discovered by the requestor. Alternatively, the requestor may request the MSMS to discover the models that may be needed by defining Inferencing Task/Data Information and Requirements. The requestor may also provide an Endpoint Profile to the MSMS and specify in the Capability Information any limitation/constraints for deployment (e.g., only a limited number of models can be deployed). For example, if the user is going to visit a zoo and may use ML models hosted on their mobile phone to perform image recognition on animals, then the corresponding requestor may discover animal image recognition related models from a model registry and provide this information to the MSMS. [00107] Step 2: The MSMS collects (real-time) information that is necessary in determining which model should be selected for deployment and for performing the inferencing task. Such information may include the analysis of the inferencing data, the status and context information of the endpoint, etc. [00108] Step 3: Based on the information collected in step 2, the MSMS may make a decision on which model should be deployed to perform the inferencing task. The decision may be made by the MSMS itself or by the decision entity. [00109] Step 4: If the selected model has not been deployed, the MSMS may instruct the deployment of the selected model, including downloading the model (inferencing pipeline) and allocating resources for the inferencing task. The MSMS may coordinate with the MSMC(s) associated with the deployment service or the endpoint management service in deploying the model. [00110] Step 5: If there are already model(s) deployed to the endpoint and deploying the newly selected model will exceed the limit of deployment, the MSMS may determine (by itself or by the decision entity) to remove one or more models or inferencing pipelines that are not in use from deployment. If the model to be removed and the newly deployed model share the pre/post-processing functions, the MSMS may determine to swap out the model only while keeping the remaining components of the inferencing pipeline. [00111] Step 6: The MSMS may update the status of the removed model and instruct the endpoint (via the corresponding MSMC) to remove the model and/or inferencing pipeline from deployment. [00112] Step 7: The MSMS may send a notification to the requestor informing the update of deployment. [00113] Steps 2 to 7 may be repeated for each inferencing data instance or each batch of inferencing data, or according to a pre-defined schedule or trigger, depending on the configurations in Notification Policy, Selection Policy and/or Management Policy. Inferencing data collection [00114] When performing inferencing tasks, the models may encounter inferencing data that is “unfamiliar” to the model. Such data may exhibit notable differences as compared with the data in the training datasets of the deployed models, or the distribution/pattern of the inferencing data deviates from any of the training dataset. As a result, the model may not perform well on such data, generating incorrect inferencing results or showing low confidence level. Usually, it is beneficial to collect such data and improve the inferencing performance by updating/re-training the model(s) or creating new model specialization with the data. [00115] Such “data of interest” or “unfamiliar data” can be identified either by data evaluation or data quality measurement performed on the inferencing data, or evaluation of the inferencing result/performance. For example, if the inferencing data shows low similarity when compared with any of the training/baseline dataset of the models, then the inferencing data may be considered as “unfamiliar”. If the inferencing result shows low confidence level or is incorrect, it may also indicate that the corresponding inferencing data is “unfamiliar”. For example, in a binary classification task, if the inferencing result indicates a 50% possibility of belonging to either class, then it could be considered as “data of interest”. [00116] The proposed MSMS may assist the procedure of identifying unfamiliar data or data of interest, collecting such data and the corresponding ground truth, and trigger model creation/update with the collected data. Figure 10 shows an example procedure. Note that the steps may occur in a different order than the one shown in the figure. [00117] Step 0: The requestor (via its corresponding MSMC) may configure a management service request with the MSMS and provide the information found in Table 4. The MSMS may also configure the endpoint according to the requirements of the request. [00118] Step 1: The MSMS may identify “data of interest” by examining the inferencing data. After the inferencing data is fed into the inferencing pipeline deployed to the endpoint, data processing or measurement may be performed by the data functions in the inferencing pipeline or other relevant services. The MSMS may request data processing or measurement results from the MSMC associated with the endpoint or the relevant services. Alternatively, the MSMS may retrieve inferencing data from the endpoint (via the corresponding MSMC), and perform processing/measurement (by itself or utilize other services) on the data. The data processing or measurement results may indicate whether the inferencing data is similar to the training/baseline dataset(s) or “unfamiliar”. [00119] Step 2: Detection of unfamiliar data may trigger a model selection as the currently deployed/selected model may not fit well. The decision can be made by the MSMS or a decision entity. [00120] Step 3: If the MSMS is not be able to determine which model should be selected for the unfamiliar data based on the Selection Policy, the MSMS may request model selection from the decision entity. For example, the Notification Policy may define a policy that the MSMS should notify the decision entity to select a model for inferencing when encounter unfamiliar data. The MSMS may send a notification to the decision entity for model selection and include data evaluation result in the notification. [00121] Step 4: The MSMS receives model selection decision from the decision entity. The decision entity may indicate the need to collect the inferencing data and the corresponding ground truth. The decision may also indicate that human review is required to replace or complement the inferencing process with the ML model(s). Alternatively, the MSMS may make the decision of model selection and data collection by itself based on the Selection Policy. [00122] Step 5: The MSMS instructs the endpoint (via the corresponding MSMC) to perform inferencing task with the selected model. [00123] Step 6: If required, the MSMS may collect the inferencing data and record the information in Inferencing data Information in the corresponding Inferencing Pipeline Profile. The data can be collected by the MSMS itself or by instructing the endpoint to perform the collection process. Particularly, if the collected inferencing data will be used to train a new model, the MSMS may create a new model profile to record the collected inferencing data (as “Training Data Information” since the data is to be used for training the new model). The Model Status can be set as “untrained”. [00124] Different “unfamiliar” data instances may not necessarily belong to the same category. In other words, the unfamiliar data instances may present different patterns that lead to the creation of multiple models. This can be identified by the MSMS when comparing an inferencing data instance with the collected inferencing data. The MSMS may create separate model profiles to record data instances with different patterns. [00125] Step 7: The MSMS may collect the inferencing results from the MSMC associated with the endpoint (by the MSMS itself or by instructing the endpoint to collect the results). The MSMS may also collect the ground truth if required by the decision entity or the Selection Policy. The ground truth may be collected from the MSMC associated with the endpoint or other relevant service (e.g., labeling service). If human review process is selected, the ground truth may be collected from the MSMC associated with the human review process. The collection of ground truth may occur at a later time when the ground truth is ready. The collected ground truth can be added to the model profile that is recording the inferencing data (as “Training Data Information” since the data is to be used for training the new model). [00126] The MSMS may evaluate the performance of the model on the inferencing data after inferencing result is generated or after the ground truth is collected, where the MSMS may identify data of interest and determines to backtrace and record the inferencing data. [00127] Step 8: When enough data of interest is collected, the MSMS may trigger the management operation and send a request to the decision entity (or the corresponding MSMC) for model creation/update. For example, the Management Policy may define that when the number of inferencing data instances collected has reached a pre-defined value, a model creation/update can be triggered. [00128] Step 9: The MSMS sends a request to train a new model or update one or more existing models with the newly collected inferencing data. The request may be sent to the MSMC associated with the requestor, the user or a training service. The request may include information of the collected inferencing data and the corresponding ground truth. [00129] Step 10: The MSMS may receive the new/updated model or model profile. RESTful API [00130] One of the approaches to realize the aforementioned ML model selection and management service functionality is through a RESTful API supported by the MSMS and MSMC. The RESTful API may comprise one or more different types of resources hosted and stored by the MSMS which are accessed by MSMC via RESTful operations commonly referred to as methods (e.g., PUT, POST, GET, DELETE). Each type of resource defined for the API may comprise a defined set of information elements commonly referred to as attributes. These resource types and their respective attributes provide a means for the MSMS to receive information from MSMC using RESTful protocols such as HTTP, process the information by performing MSMS specific tasks, and provide results of these operations back to the MSMC. Hence the RESTful APIs are used by MSMC to interface with the MSMS and access the supported capabilities of the MSMS in a RESTful manner. [00131] The RESTful API of the MSMS may support an Endpoint Profile resource type. This Endpoint Profile resource may comprise attributes such as but not limited to those defined in Table 1. [00132] The RESTful API of the MSMS may also support an Inferencing Pipeline Profile resource type. This Inferencing Pipeline Profile resource may comprise attributes such as but not limited to the information elements defined in Table 2. [00133] The RESTful API of the MSMS may also support a Model Profile resource type. This Model Profile resource may comprise attributes such as but not limited to those defined in Table 3. [00134] The RESTful API of the MSMS may also support a Service Configuration resource type. This Service Configuration resource may comprise attributes such as but not limited to the information elements defined in Table 4. [00135] The RESTful API of the MSMS may also support additional resource types. For example, the MSMS may support a resource type which is used by MSMS to retrieve ML models from a model registry or repository, a resource type which is used by MSMS to request for data collection (inferencing data, inferencing results, or ground truth), a resource type which is used by MSMS or MSMC to request for model training/update/re- training from a training service. These resources may be separate from the aforementioned resource types. Graphical User Interface [00136] The following figures show an example GUI for the model selection and management service (MSMS) proposed in this disclosure. A user (requestor) may use the GUI to request the service, provide information needed by the service, and view information (result) of the service. [00137] Figure 11 shows the GUI for service overview. The user may choose different options by clicking on the corresponding button to view and/or edit information regarding the endpoints, the ML models, the inferencing pipeline, to configure policies for notification, model selection and management operations. [00138] Figure 12 shows the GUI for viewing and editing endpoint information. The user may provide endpoint information to the MSMS by adding endpoint and specifying information of the endpoint, which corresponds to the procedure where the requestor may specify endpoint information or endpoint profiles in the management service request. The GUI may also display information of the endpoints that have been added, where the displayed information is based on the endpoint profiles maintained at the MSMS. [00139] A similar GUI can also be provided for the user to view and edit information of the ML models and inferencing pipelines. Alternatively, a GUI as in Figure 13 can be provided to visualize the deployment of models and inferencing pipelines. [00140] With the GUI defined in Figure 13, the user may provide model and inferencing pipeline information to the MSMS, which corresponds to the procedure where the requestor may specify model and pipeline information or profiles in the management service request. Particularly, the user may drag and drop models and processing logics to edit the pipeline and specify its topology. The user may click on a model in the pipeline to view or edit information of the model, as well as to manage the model, such as to select/deselect, update, activate/deactivate, remove the model. The user may also click on a data instance (e.g., inference data, inference result, intermediate results) in the pipeline to view the information of the data, such as its characteristics, quality measurements/evaluations, storage location, etc. [00141] Figure 14 shows the GUI for viewing and editing policies. The user may add new policies by specifying the policy type (notification, selection, management), policy target (notification target, decision entity), policy condition (when a notification should be generated, when a model selection or management operation should be performed), expected action (sending notification, performing model selection or management operation, collecting data, updating deployment, etc.) and other information (information to be included in the notification, model selection criteria), which corresponds to the procedure where the requestor may specify Notification Policy, Selection Policy and Management Policy in the management service request. The user may choose to apply default policies, or select from options provided by the MSMS, or define customized policies. The GUI also allows the user to edit or remove an existing policy. Example Communications System [00142] The 3rd Generation Partnership Project (3GPP) develops technical standards for cellular telecommunications network technologies, including radio access, the core transport network, and service capabilities - including work on codecs, security, and quality of service. Recent radio access technology (RAT) standards include WCDMA (commonly referred as 3G), LTE (commonly referred as 4G), LTE-Advanced standards, and New Radio (NR), which is also referred to as “8G”.3GPP NR standards development is expected to continue and include the definition of next generation radio access technology (new RAT), which is expected to include the provision of new flexible radio access below 7 GHz, and the provision of new ultra-mobile broadband radio access above 7 GHz. The flexible radio access is expected to consist of a new, non-backwards compatible radio access in new spectrum below 7 GHz, and it is expected to include different operating modes that may be multiplexed together in the same spectrum to address a broad set of 3GPP NR use cases with diverging requirements. The ultra-mobile broadband is expected to include cmWave and mmWave spectrum that will provide the opportunity for ultra-mobile broadband access for, e.g., indoor applications and hotspots. In particular, the ultra-mobile broadband is expected to share a common design framework with the flexible radio access below 7 GHz, with cmWave and mmWave specific design optimizations. [00143] 3GPP has identified a variety of use cases that NR is expected to support, resulting in a wide variety of user experience requirements for data rate, latency, and mobility. The use cases include the following general categories: enhanced mobile broadband (eMBB) ultra-reliable low-latency Communication (URLLC), massive machine type communications (mMTC), network operation (e.g., network slicing, routing, migration and interworking, energy savings), and enhanced vehicle-to-everything (eV2X) communications, which may include any of Vehicle-to-Vehicle Communication (V2V), Vehicle-to- Infrastructure Communication (V2I), Vehicle-to-Network Communication (V2N), Vehicle- to-Pedestrian Communication (V2P), and vehicle communications with other entities. Specific service and applications in these categories include, e.g., monitoring and sensor networks, device remote controlling, bi-directional remote controlling, personal cloud computing, video streaming, wireless cloud-based office, first responder connectivity, automotive ecall, disaster alerts, real-time gaming, multi-person video calls, autonomous driving, augmented reality, tactile internet, virtual reality, home automation, robotics, and aerial drones to name a few. All of these use cases and others are contemplated herein. [00144] Figure 15A illustrates an example communications system 100 in which the methods and apparatuses described and claimed herein may be an aspect of. As shown, the example communications system 100 may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, 102e, 102f, and/or 102g (which generally or collectively may be referred to as WTRU 102), a radio access network (RAN) 103/104/105/103b/104b/108B, a core network 106/107/109, a public switched telephone network (PSTN) 108, the Internet 110, other networks 112, and V2X server (or ProSe function and server) 113, though it will be appreciated that the disclosed examples contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d, 102e, 102f, 102g may be any type of apparatus or device configured to operate and/or communicate in a wireless environment. Although each WTRU 102a, 102b, 102c, 102d, 102e, 102f, 102g is depicted in Figures 9A-9E as a hand-held wireless communications apparatus, it is understood that with the wide variety of use cases contemplated for 8G wireless communications, each WTRU may comprise or be embodied in any type of apparatus or device configured to transmit and/or receive wireless signals, including, by way of example only, user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a tablet, a netbook, a notebook computer, a personal computer, a wireless sensor, consumer electronics, a wearable device such as a smart watch or smart clothing, a medical or eHealth device, a robot, industrial equipment, a drone, a vehicle such as a car, truck, train, or airplane, and the like. [00145] The communications system 100 may also include a base station 114a and a base station 114b. Base stations 114a may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, and/or the other networks 112. Base stations 114b may be any type of device configured to wiredly and/or wirelessly interface with at least one of the RRHs (Remote Radio Heads) 118a, 118b, TRPs (Transmission and Reception Points) 119a, 119b, and/or RSUs (Roadside Units) 120a and 120b to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, the other networks 112, and/or V2X server (or ProSe function and server) 113. RRHs 118a, 118b may be any type of device configured to wirelessly interface with at least one of the WTRU 102c, to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, and/or the other networks 112. TRPs 119a, 119b may be any type of device configured to wirelessly interface with at least one of the WTRU 102d, to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, and/or the other networks 112. RSUs 120a and 120b may be any type of device configured to wirelessly interface with at least one of the WTRU 102e or 102f, to facilitate access to one or more communication networks, such as the core network 106/107/109, the Internet 110, the other networks 112, and/or V2X server (or ProSe function and server) 113. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements. [00146] The base station 114a may be part of the RAN 103/104/105, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114b may be part of the RAN 103b/104b/108B, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a may be configured to transmit and/or receive wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The base station 114b may be configured to transmit and/or receive wired and/or wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, the base station 114a may include three transceivers, e.g., one for each sector of the cell. In some cases, the base station 114a may employ multiple-input multiple output (MIMO) technology and, therefore, may utilize multiple transceivers for each sector of the cell. [00147] The base stations 114a may communicate with one or more of the WTRUs 102a, 102b, 102c over an air interface 115/116/117, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, cmWave, mmWave, etc.). The air interface 115/116/117 may be established using any suitable radio access technology (RAT). [00148] The base stations 114b may communicate with one or more of the RRHs 118a, 118b, TRPs 119a, 119b, and/or RSUs 120a and 120b, over a wired or air interface 118B/116b/117b, which may be any suitable wired (e.g., cable, optical fiber, etc.) or wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, cmWave, mmWave, etc.). The air interface 118B/116b/117b may be established using any suitable radio access technology (RAT). [00149] The RRHs 118a, 118b, TRPs 119a, 119b and/or RSUs 120a, 120b, may communicate with one or more of the WTRUs 102c, 102d, 102e, 102f over an air interface 118C/116c/117c, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, cmWave, mmWave, etc.). The air interface 118C/116c/117c may be established using any suitable radio access technology (RAT). [00150] The WTRUs 102a, 102b, 102c,102d, 102e, 102f, and/or 102g may communicate with one another over an air interface 118D/116d/117d (not shown in the figures), which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, cmWave, mmWave, etc.). The air interface 118D/116d/117d may be established using any suitable radio access technology (RAT). [00151] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 103/104/105 and the WTRUs 102a, 102b, 102c, or RRHs 118a, 118b,TRPs 119a, 119b and RSUs 120a, 120b, in the RAN 103b/104b/108B and the WTRUs 102c, 102d, 102e, 102f, may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115/116/117 or 118C/116c/117c respectively using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink Packet Access (HSDPA) and/or High-Speed Uplink Packet Access (HSUPA). [00152] In some cases, the base station 114a and the WTRUs 102a, 102b, 102c, or RRHs 118a, 118b, TRPs 119a, 119b, and/or RSUs 120a, 120b, in the RAN 103b/104b/108B and the WTRUs 102c, 102d, may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 115/116/117 or 118C/116c/117c respectively using Long Term Evolution (LTE) and/or LTE-Advanced (LTE-A). In the future, the air interface 115/116/117 may implement 3GPP NR technology. The LTE and LTE-A technology includes LTE D2D and V2X technologies and interface (such as Sidelink communications, etc.) The 3GPP NR technology includes NR V2X technologies and interface (such as Sidelink communications, etc.) [00153] In some cases, the base station 114a in the RAN 103/104/105 and the WTRUs 102a, 102b, 102c, or RRHs 118a, 118b, TRPs 119a, 119b and/or RSUs 120a, 120b, in the RAN 103b/104b/108B and the WTRUs 102c, 102d, 102e, 102f may implement radio technologies such as IEEE 802.16 (e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA20001X, CDMA2000 EV-DO, Interim Standard 2000 (IS- 2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like. [00154] The base station 114c in Figure 15A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, and the like. In some cases, the base station 114c and the WTRUs 102e, may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In some cases, the base station 114c and the WTRUs 102d, may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In some cases, the base station 114c and the WTRUs 102e, may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell or femtocell. As shown in Figure 15A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114c may not be required to access the Internet 110 via the core network 106/107/109. [00155] The RAN 103/104/105 and/or RAN 103b/104b/108B may be in communication with the core network 106/107/109, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. For example, the core network 106/107/109 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high- level security functions, such as user authentication. [00156] Although not shown in Figure 15A, it will be appreciated that the RAN 103/104/105 and/or RAN 103b/104b/108B and/or the core network 106/107/109 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 103/104/105 and/or RAN 103b/104b/108B or a different RAT. For example, in addition to being connected to the RAN 103/104/105 and/or RAN 103b/104b/108B, which may be utilizing an E-UTRA radio technology, the core network 106/107/109 may also be in communication with another RAN (not shown) employing a GSM radio technology. [00157] The core network 106/107/109 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d, 102e to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 112 may include wired or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another core network connected to one or more RANs, which may employ the same RAT as the RAN 103/104/105 and/or RAN 103b/104b/108B or a different RAT. [00158] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities, e.g., the WTRUs 102a, 102b, 102c, 102d, and 102e may include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102e shown in Figure 15A may be configured to communicate with the base station 114a, which may employ a cellular- based radio technology, and with the base station 114c, which may employ an IEEE 802 radio technology. [00159] Figure 15B is a block diagram of an example apparatus or device configured for wireless communications in accordance with the aspects illustrated herein, such as for example, a WTRU 102. As shown in Figure 15B, the example WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 113, a display/touchpad/indicators 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an example. Also, in some cases the base stations 114a and 114b, and/or the nodes that base stations 114a and 114b may represent, such as but not limited to transceiver station (BTS), a Node-B, a site controller, an access point (AP), a home node-B, an evolved home node-B (eNodeB), a home evolved node-B (HeNB), a home evolved node-B gateway, and proxy nodes, among others, may include some or all of the elements depicted in Figure 15B and described herein. [00160] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While Figure 15B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip. [00161] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 115/116/117. For example, in some cases, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In some cases, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In some cases, the transmit/receive element 122 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals. [00162] In addition, although the transmit/receive element 122 is depicted in Figure 15B as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in some cases, the WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 115/116/117. [00163] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11, for example. [00164] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad/indicators 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad/indicators 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In some cases, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown). [00165] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries, solar cells, fuel cells, and the like. [00166] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 115/116/117 from a base station (e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an aspect. [00167] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include various sensors such as an accelerometer, biometrics (e.g., finger print) sensors, an e- compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port or other interconnect interfaces, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like. [00168] The WTRU 102 may be embodied in other apparatuses or devices, such as a sensor, consumer electronics, a wearable device such as a smart watch or smart clothing, a medical or eHealth device, a robot, industrial equipment, a drone, a vehicle such as a car, truck, train, or airplane. The WTRU 102 may connect to other components, modules, or systems of such apparatuses or devices via one or more interconnect interfaces, such as an interconnect interface that may comprise one of the peripherals 138. [00169] Figure 15C is a system diagram of the RAN 103 and the core network 106. As noted above, the RAN 103 may employ a UTRA radio technology to communicate with the WTRUs 102a, 102b, and 102c over the air interface 115. The RAN 103 may also be in communication with the core network 106. As shown in Figure 15C, the RAN 103 may include Node-Bs 140a, 140b, 140c, which may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 115. The Node-Bs 140a, 140b, 140c may each be associated with a particular cell (not shown) within the RAN 103. The RAN 103 may also include RNCs 142a, 142b. It will be appreciated that the RAN 103 may include any number of Node-Bs and RNCs while remaining consistent with an aspect of the disclosure. [00170] As shown in Figure 15C, the Node-Bs 140a, 140b may be in communication with the RNC 142a. Additionally, the Node-B 140c may be in communication with the RNC 142b. The Node-Bs 140a, 140b, 140c may communicate with the respective RNCs 142a, 142b via an Iub interface. The RNCs 142a, 142b may be in communication with one another via an Iur interface. Each of the RNCs 142a, 142b may be configured to control the respective Node-Bs 140a, 140b, 140c to which it is connected. In addition, each of the RNCs 142a, 142b may be configured to carry out or support other functionality, such as outer loop power control, load control, admission control, packet scheduling, handover control, macro-diversity, security functions, data encryption, and the like. [00171] The core network 106 shown in Figure 15C may include a media gateway (MGW) 144, a mobile switching center (MSC) 146, a serving GPRS support node (SGSN) 148, and/or a gateway GPRS support node (GGSN) 150. While each of the foregoing elements are depicted as part of the core network 106, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator. [00172] The RNC 142a in the RAN 103 may be connected to the MSC 146 in the core network 106 via an IuCS interface. The MSC 146 may be connected to the MGW 144. The MSC 146 and the MGW 144 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. [00173] The RNC 142a in the RAN 103 may also be connected to the SGSN 148 in the core network 106 via an IuPS interface. The SGSN 148 may be connected to the GGSN 150. The SGSN 148 and the GGSN 150 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between and the WTRUs 102a, 102b, 102c and IP-enabled devices. [00174] As noted above, the core network 106 may also be connected to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers. [00175] Figure 15D is a system diagram of the RAN 104 and the core network 107. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, and 102c over the air interface 116. The RAN 104 may also be in communication with the core network 107. [00176] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an aspect of the disclosure. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In some cases, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a. [00177] Each of the eNode-Bs 160a, 160b, and 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the uplink and/or downlink, and the like. As shown in Figure 15D, the eNode-Bs 160a, 160b, 160c may communicate with one another over an X2 interface. [00178] The core network 107 shown in Figure 15D may include a mobility management gateway (MME) 162, a serving gateway 164, and a packet data network (PDN) gateway 166. While each of the foregoing elements are depicted as part of the core network 107, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator. [00179] The MME 162 may be connected to each of the eNode-Bs 160a, 160b, and 160c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may also provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM or WCDMA. [00180] The serving gateway 164 may be connected to each of the eNode-Bs 160a, 160b, and 160c in the RAN 104 via the S1 interface. The serving gateway 164 may generally route and forward user data packets to/from the WTRUs 102a, 102b, 102c. The serving gateway 164 may also perform other functions, such as anchoring user planes during inter- eNode B handovers, triggering paging when downlink data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like. [00181] The serving gateway 164 may also be connected to the PDN gateway 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. [00182] The core network 107 may facilitate communications with other networks. For example, the core network 107 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the core network 107 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the core network 107 and the PSTN 108. In addition, the core network 107 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers. [00183] Figure 15E is a system diagram of the RAN 105 and the core network 109. The RAN 105 may be an access service network (ASN) that employs IEEE 802.16 radio technology to communicate with the WTRUs 102a, 102b, and 102c over the air interface 117. As will be further discussed below, the communication links between the different functional entities of the WTRUs 102a, 102b, 102c, the RAN 105, and the core network 109 may be defined as reference points. [00184] As shown in Figure 15E, the RAN 105 may include base stations 180a, 180b, 180c, and an ASN gateway 182, though it will be appreciated that the RAN 105 may include any number of base stations and ASN gateways while remaining consistent with an aspect of the disclosure. The base stations 180a, 180b, 180c may each be associated with a particular cell in the RAN 105 and may include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 117. In some cases, the base stations 180a, 180b, 180c may implement MIMO technology. Thus, the base station 180a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a. The base stations 180a, 180b, 180c may also provide mobility management functions, such as handoff triggering, tunnel establishment, radio resource management, traffic classification, quality of service (QoS) policy enforcement, and the like. The ASN gateway 182 may serve as a traffic aggregation point and may be responsible for paging, caching of subscriber profiles, routing to the core network 109, and the like. [00185] The air interface 117 between the WTRUs 102a, 102b, 102c and the RAN 105 may be defined as an R1 reference point that implements the IEEE 802.16 specification. In addition, each of the WTRUs 102a, 102b, and 102c may establish a logical interface (not shown) with the core network 109. The logical interface between the WTRUs 102a, 102b, 102c and the core network 109 may be defined as an R2 reference point, which may be used for authentication, authorization, IP host configuration management, and/or mobility management. [00186] The communication link between each of the base stations 180a, 180b, and 180c may be defined as an R8 reference point that includes protocols for facilitating WTRU handovers and the transfer of data between base stations. The communication link between the base stations 180a, 180b, 180c and the ASN gateway 182 may be defined as an R6 reference point. The R6 reference point may include protocols for facilitating mobility management based on mobility events associated with each of the WTRUs 102a, 102b, 102c. [00187] As shown in Figure 15E, the RAN 105 may be connected to the core network 109. The communication link between the RAN 105 and the core network 109 may defined as an R3 reference point that includes protocols for facilitating data transfer and mobility management capabilities, for example. The core network 109 may include a mobile IP home agent (MIP-HA) 184, an authentication, authorization, accounting (AAA) server 186, and a gateway 188. While each of the foregoing elements are depicted as part of the core network 109, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator. [00188] The MIP-HA may be responsible for IP address management, and may enable the WTRUs 102a, 102b, and 102c to roam between different ASNs and/or different core networks. The MIP-HA 184 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The AAA server 186 may be responsible for user authentication and for supporting user services. The gateway 188 may facilitate interworking with other networks. For example, the gateway 188 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. In addition, the gateway 188 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers. [00189] Although not shown in Figure 15E, it will be appreciated that the RAN 105 may be connected to other ASNs and the core network 109 may be connected to other core networks. The communication link between the RAN 105 the other ASNs may be defined as an R4 reference point, which may include protocols for coordinating the mobility of the WTRUs 102a, 102b, 102c between the RAN 105 and the other ASNs. The communication link between the core network 109 and the other core networks may be defined as an R5 reference, which may include protocols for facilitating interworking between home core networks and visited core networks. [00190] The core network entities described herein and illustrated in Figures 9A, 9C, 9D, and 9E are identified by the names given to those entities in certain existing 3GPP specifications, but it is understood that in the future those entities and functionalities may be identified by other names and certain entities or functions may be combined in future specifications published by 3GPP, including future 3GPP NR specifications. Thus, the particular network entities and functionalities described and illustrated in Figures 9A, 9B, 9C, 9D, and 9E are provided by way of example only, and it is understood that the subject matter disclosed and claimed herein may be embodied or implemented in any similar communication system, whether presently defined or defined in the future. [00191] Figure 15F is a block diagram of an exemplary computing system 90 in which one or more apparatuses of the communications networks illustrated in Figures 9A, 9C, 9D and 9E may be embodied, such as certain nodes or functional entities in the RAN 103/104/105, Core Network 106/107/109, PSTN 108, Internet 110, or Other Networks 112. Computing system 90 may comprise a computer or server and may be controlled primarily by computer readable instructions, which may be in the form of software, wherever, or by whatever means such software is stored or accessed. Such computer readable instructions may be executed within a processor 91, to cause computing system 90 to do work. The processor 91 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 91 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the computing system 90 to operate in a communications network. Coprocessor 81 is an optional processor, distinct from main processor 91, that may perform additional functions or assist processor 91. Processor 91 and/or coprocessor 81 may receive, generate, and process data related to the methods and apparatuses disclosed herein. [00192] In operation, processor 91 fetches, decodes, and executes instructions, and transfers information to and from other resources via the computing system’s main data- transfer path, system bus 80. Such a system bus connects the components in computing system 90 and defines the medium for data exchange. System bus 80 typically includes data lines for sending data, address lines for sending addresses, and control lines for sending interrupts and for operating the system bus. An example of such a system bus 80 is the PCI (Peripheral Component Interconnect) bus. [00193] Memories coupled to system bus 80 include random access memory (RAM) 82 and read only memory (ROM) 93. Such memories include circuitry that allows information to be stored and retrieved. ROMs 93 generally contain stored data that cannot easily be modified. Data stored in RAM 82 may be read or changed by processor 91 or other hardware devices. Access to RAM 82 and/or ROM 93 may be controlled by memory controller 92. Memory controller 92 may provide an address translation function that translates virtual addresses into physical addresses as instructions are executed. Memory controller 92 may also provide a memory protection function that isolates processes within the system and isolates system processes from user processes. Thus, a program running in a first mode may access only memory mapped by its own process virtual address space; it cannot access memory within another process’s virtual address space unless memory sharing between the processes has been set up. [00194] In addition, computing system 90 may contain peripherals controller 83 responsible for communicating instructions from processor 91 to peripherals, such as printer 94, keyboard 84, mouse 95, and disk drive 85. [00195] Display 86, which is controlled by display controller 96, is used to display visual output generated by computing system 90. Such visual output may include text, graphics, animated graphics, and video. The visual output may be provided in the form of a graphical user interface (GUI). Display 86 may be implemented with a CRT-based video display, an LCD-based flat-panel display, gas plasma-based flat-panel display, or a touch- panel. Display controller 96 includes electronic components required to generate a video signal that is sent to display 86. [00196] Further, computing system 90 may contain communication circuitry, such as for example a network adapter 97, that may be used to connect computing system 90 to an external communications network, such as the RAN 103/104/105, Core Network 106/107/109, PSTN 108, Internet 110, or Other Networks 112 of Figures 9A, 9B, 9C, 9D, and 9E, to enable the computing system 90 to communicate with other nodes or functional entities of those networks. The communication circuitry, alone or in combination with the processor 91, may be used to perform the transmitting and receiving steps of certain apparatuses, nodes, or functional entities described herein. [00197] Figure 15G illustrates an example communications system 111 in which the methods and apparatuses described and claimed herein may be an aspect of. As shown, the example communications system 111 may include wireless transmit/receive units (WTRUs) A, B, C, D, E, F, a base station, a V2X server, and a RSUs A and B, though it will be appreciated that the disclosure contemplates any number of WTRUs, base stations, networks, and/or network elements. One or several or all WTRUs A, B, C, D, E can be out of range of the network (for example, in the figure out of the cell coverage boundary shown as the dash line). WTRUs A, B, C form a V2X group, among which WTRU A is the group lead and WTRUs B and C are group members. WTRUs A, B, C, D, E, F may communicate over Uu interface or Sidelink (PC5) interface. [00198] It is understood that any or all of the apparatuses, systems, methods and processes described herein may be embodied in the form of computer executable instructions (e.g., program code) stored on a computer-readable storage medium which instructions, when executed by a processor, such as processors 118 or 91, cause the processor to perform and/or implement the systems, methods and processes described herein. Specifically, any of the steps, operations or functions described herein may be implemented in the form of such computer executable instructions, executing on the processor of an apparatus or computing system configured for wireless and/or wired network communications. Computer readable storage media include volatile and nonvolatile, removable and non-removable media implemented in any non-transitory (e.g., tangible or physical) method or technology for storage of information, but such computer readable storage media do not include signals. Computer readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible or physical medium which may be used to store the desired information and which may be accessed by a computing system. Definitions [00199] Provided below are definitions for abbreviations found within the body of the disclosure. AI/ML Artificial Intelligence / Machine Learning API Application Programming Interface GUI Graphical User Interface IoT Internet of Things MSMC Model Selection and Management Client MSMS Model Selection and Management Service URI Uniform Resource Identifier URL Uniform Resource Locator [00200] Provided below are definitions for terms found within the body of the disclosure. M_{odel deployment} ^{Model deployment is the process of deploying a trained ML model to} an endpoint to perform inferencing tasks according to the user’s request. E_ndpoint ^{Endpoint is the entity that provides an interface for a ML user to use} the trained ML models for inferencing tasks. Inferencing When deploying a model, the corresponding pre/post-processing pipeline functions can be deployed together with the model, as an inferencing pipeline. I_{nferencing task} ^{An inferencing task is defined as the process of taking in one or more} inferencing data instances, performing inferencing on the data with the ML models or pipelines, and generating inferencing result. I_{nferencing data} ^{Inferencing data is the input data to an inferencing pipeline. The} inferencing data will be fed into the deployed model(s) to generate inferencing result. I_{nferencing result} ^{Inferencing result is the output of an inferencing pipeline that is} generated with the deployed ML models/pipelines. _{Ground truth} ^{The true target value associated with the inferencing data in an} inferencing task, such as the true label of the data, the actual value of a prediction target, etc. Ground truth may be used for evaluating the inferencing performance (i.e., whether the inferencing result is correct/accurate) and be combined with the inferencing data for the training of a new/updated model. _User ^{The ML user is the entity that initiates the request for a ML service or} an inferencing task. The user could be a human or an application/service that may invoke the ML service/model to perform inferencing. _Requestor ^{The Requestor is the entity that initiates the request for the model} selection and management service. The requestor can be the user, the endpoint, or a relevant service (e.g., ML service, model deployment service, endpoint management service).

Claims

What is claimed: 1. A method performed by a first service entity for a service supporting Machine Learning (ML) model distribution, the method comprising: receiving, from a second entity associated with a user device, a request to select a ML model for distribution, wherein the request comprises information indicative of one or more ML models to be distributed to the user device; receiving information corresponding to the user device; selecting, based at least in part on the request and the information of the user device, a first ML model to be distributed to the user device; sending, to the second entity, information indicative of the selected ML model; receiving updated information of the user device or the one or more ML models; selecting, based at least in part on the updated information and the request, a second ML model to be distributed to the user device; and sending, to the second entity, information of the second selected ML model.

2. The method of claim 1, wherein the information of the user device comprises location information of the user device.

3. The method of claim 1, wherein the information of the user device comprises capability information of the user device.

4. The method of claim 1, wherein the request further comprises policies for selecting ML model for distribution.

5. The method of claim 1, further comprising: sending, to the second entity, a notification to remove the first selected ML model from the user device.

6. The method of claim 1, further comprising: deploying the first selected ML model to the second entity.

7. The method of claim 1, further comprising: determining, based on the updated information, a trigger for switching to another ML model is met, wherein the selecting the second ML model is based on the determining.

8. An apparatus supporting Machine Learning (ML) model distribution, the apparatus comprising: one or more processors; and memory storing a set of instructions, that, when executed by the one or more processors, cause: receiving, from an entity associated with a user device, a request to select a ML model for distribution, wherein the request comprises information indicative of one or more ML models to be distributed to the user device; receiving information corresponding to the user device; selecting, based at least in part on the request and the information of the user device, a first ML model to be distributed to the user device; sending, to the entity, information indicative of the selected ML model; receiving, updated information of the user device or the one or more ML models; selecting, based at least in part on the updated information and the request, a second ML model to be distributed to the user device; and sending, to the entity, information of the second selected ML model.

9. The apparatus of claim 8, wherein the information of the user device comprises location information of the user device.

10. The apparatus of claim 8, wherein the information of the user device comprises capability information of the user device.

11. The apparatus of claim 8, wherein the request further comprises policies for selecting ML model for distribution.

12. The apparatus of claim 8, wherein the set of instructions, when executed by the one or more processors, further cause: sending, to the entity, a notification to remove the first selected ML model from the user device.

13. The apparatus of claim 8, wherein the set of instructions, when executed by the one or more processors, further cause: deploying the first selected ML model to the entity.

14. The apparatus of claim 1, wherein the set of instructions, when executed by the one or more processors, further cause: determining, based on the updated information, a trigger for switching to another ML model is met, wherein the selecting the second ML model is based on the determining.

15. A non-transitory computer-readable medium supporting Machine Learning (ML) model distribution, the non-transitory computer-readable medium comprising a set of computer-executable instructions, that, when executed by one or more processors, cause: receiving, from an entity associated with a user device, a request to select a ML model for distribution, wherein the request comprises information indicative of one or more ML models to be distributed to the user device; receiving information corresponding to the user device; selecting, based at least in part on the request and the information of the user device, a first ML model to be distributed to the user device; sending, to the entity, information indicative of the selected ML model; receiving, updated information of the user device or the one or more ML models; selecting, based at least in part on the updated information and the request, a second ML model to be distributed to the user device; and sending, to the entity, information of the second selected ML model.

16. The non-transitory computer-readable medium of claim 15, wherein the information of the user device comprises location information of the user device.

17. The non-transitory computer-readable medium of claim 15, wherein the information of the user device comprises capability information of the user device.

18. The non-transitory computer-readable medium of claim 15, wherein the request further comprises policies for selecting ML model for distribution.

19. The non-transitory computer-readable medium of claim 15, wherein the set of instructions, when executed by the one or more processors, further cause: sending, to the entity, a notification to remove the first selected ML model from the user device.

20. The non-transitory computer-readable medium of claim 15, wherein the set of instructions, when executed by the one or more processors, further cause: deploying the first selected ML model to the entity.