KR20240125421A - Method for providing a model for analytics of network data and devices for performing the same - Google Patents

Abstract

A method for providing a model for analyzing network data and devices for performing the same are disclosed. An operation method of a provider providing an ML model according to one embodiment may include an operation of providing a first ML model to a consumer, an operation of collecting data from data sources to evaluate the first model, and an operation of providing a second ML model to the consumer based on an evaluation result of the first model.

Description

{METHOD FOR PROVIDING A MODEL FOR ANALYTICS OF NETWORK DATA AND DEVICES FOR PERFORMING THE SAME}

The disclosure below relates to a method for providing a model for analyzing network data and devices for performing the same.

The 5G mobile communication system has defined the Network Data Analytics Function (NWDAF), a network function that provides the ability to analyze and provide data collected from the 5G network to support network automation.

To automate and optimize the 5G mobile communication system, NWDAF collects raw data of each network function and application function, converts it into big data, and processes the big data to provide network analysis information.

NWDAF can consider machine learning models to derive network analysis information.

The background technology described above is technology that the inventor possessed or acquired in the process of deriving the disclosure content of the present application, and cannot necessarily be said to be a publicly known technology disclosed to the general public prior to the present application.

A method of operation of a provider providing an ML model according to one embodiment may include providing a first ML model to a consumer, collecting data from data sources to evaluate the first model, and providing a second ML model to the consumer based on a result of the evaluation of the first model.

The above provider may be a network data analytics function (NWDAF) including a Model Training logical function (MTLF), and the above consumer may be a NWDAF including an Analytics logical function (AnLF).

The method further comprises an action of registering the consumer's use of the ML model, wherein the registration may indicate a capability to convey one or more of analysis feedback information for an analysis generated by the ML model and information about analysis accuracy of the ML model.

The above evaluating operation may include an operation of collecting data from data sources NF and DCCF and monitoring the accuracy of the first ML model.

The above evaluating operation may include an operation of monitoring the accuracy of the first ML model by collecting ADRF data using at least one of an ADRF ID and a DataSetTag.

The above evaluating operation may include an operation of collecting fault prediction analytics data from the MDAS to monitor the accuracy of the first ML model.

The above evaluating operation includes an operation of subscribing to UDM to be notified of changes in subscription data for the target of the first ML model by calling Nudm_SDM_Subscribe service operation, and the UDM can subscribe to UDR to be notified of UE subscription data by calling Nudr_DM_Subscribe service operation.

The second ML model may be a first ML model or a newly generated ML model retrained based on data collected from the data sources.

A server device providing an ML model according to one embodiment includes a processor, and a memory electrically connected to the processor and storing instructions executable by the processor, wherein when the instructions are executed by the processor, the processor performs a plurality of operations, and the plurality of operations may include an operation of providing a first ML model to a consumer, an operation of collecting data from data sources to evaluate the first model, and an operation of providing a second ML model to the consumer based on a result of the evaluation of the first model.

The above server device may be a network data analytics function (NWDAF) including a Model Training logical function (MTLF), and the consumer may be a NWDAF including an Analytics logical function (AnLF).

The above plurality of actions may further include an action of registering the consumer's use of the ML model, wherein the registration may indicate a capability to convey one or more of analysis feedback information for an analysis generated by the ML model and information about analysis accuracy of the ML model.

The above evaluating operation may include an operation of collecting data from data sources NF and DCCF and monitoring the accuracy of the first ML model.

The above evaluating operation may include an operation of monitoring the accuracy of the first ML model by collecting ADRF data using at least one of an ADRF ID and a DataSetTag.

The above evaluating operation may include an operation of collecting fault prediction analytics data from the MDAS to monitor the accuracy of the first ML model.

The above evaluating operation includes an operation of subscribing to UDM to be notified of changes in subscription data for the target of the first ML model by calling Nudm_SDM_Subscribe service operation, and the UDM can subscribe to UDR to be notified of UE subscription data by calling Nudr_DM_Subscribe service operation.

The second ML model may be a first ML model or a newly generated ML model retrained based on data collected from the data sources.

Figure 1 illustrates a network system according to one embodiment.
FIG. 2 is a diagram for explaining a network data analysis process according to one embodiment.
FIG. 3 is a diagram for explaining the operation of NWDAF according to one embodiment.
FIG. 4 is a drawing for explaining the structure of NWDAF according to one embodiment.
FIG. 5 is a diagram illustrating an example of a procedure for analysis subscription/unsubscription according to one embodiment.
FIG. 6 is a diagram illustrating another example of a procedure for analysis subscription/unsubscription according to one embodiment.
FIG. 6 is a diagram illustrating another example of a procedure for analysis subscription/unsubscription according to one embodiment.
Figure 7 illustrates an example of a ML model accuracy monitoring procedure according to one embodiment.
FIG. 8 illustrates a schematic block diagram of a device performing NWDAF according to one embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be implemented in various forms. Therefore, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present disclosure includes modifications, equivalents, or alternatives included in the technical idea described in the embodiments.

Although the terms first or second may be used to describe various components, such terms should be construed only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or any and all possible combinations thereof. It should be understood that, as used herein, the terms "comprises" or "have" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

The term "module" as used in this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component that is configured integrally or may be a minimum unit of the component that performs one or more functions, or a portion thereof. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

The term '~part' used in this document means a software or hardware component such as an FPGA or ASIC, and the '~part' performs certain roles. However, the '~part' is not limited to software or hardware. The '~part' may be configured to be on an addressable storage medium and may be configured to regenerate one or more processors. For example, the '~part' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided in the components and '~parts' may be combined into a smaller number of components and '~parts' or further separated into additional components and '~parts'. In addition, the components and '~parts' may be implemented to regenerate one or more CPUs in a device or a secure multimedia card. Additionally, '~bu' may include one or more processors.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. When describing with reference to the attached drawings, identical components are given the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

The terms used in the description of this specification to identify connection nodes, terms referring to network entities, terms referring to messages, terms referring to interfaces between network entities, terms referring to various identification information, etc. are provided as examples for convenience of explanation. Therefore, the present invention is not limited to the terms described below in this specification, and other terms referring to objects having equivalent technical meanings may be used.

In this specification, for the convenience of explanation, the terms and names defined in the LTE and NR standards, which are the latest standards defined by the 3rd Generation Partnership Project (3GPP) organization among the existing communication standards, are used. However, the embodiment described below is not limited by the common terms and names, and can be equally applied to systems that follow other standards.

Figure 1 illustrates a network system according to one embodiment.

Referring to FIG. 1, according to one embodiment, a network system (10) (e.g., a 5G network system) may include a plurality of entities (100 to 190). A user equipment (UE) (100) may access a 5G core network through a radio access network (RAN) (110). The RAN (110) may refer to a base station that provides a wireless communication function to the UE (100). An Operation, Administration, and Maintenance (OAM) (190) may be a system for managing a terminal and a network.

The unit performed by each function provided by the network system (10) can be defined as a network function (NF). The NF can include an access and mobility management function (AMF) (120), a session management function (SMF) (130), a user plane function (UPF) (140), an application function (AF) (150), a policy control function (PCF) (160), a network repository function (NRF) (170), a network exposure function (NEF) (175), a messaging framework adaptor function (MFAF) (177), a network data analytics function (NWDAF) (180), a data collection coordination function (DCCF) (185), an analytics data repository function (ADRF) (187), a unified data repository (UDR) (188), and a unified data management (UDM) (189). AMF (120) manages network access and mobility of the terminal, SMF (130) performs functions related to sessions, UPF (140) is responsible for transferring user data, and AF (150) can perform a role of communicating with 5GC to provide application services. PCF (160) manages policies, and NRF (170) can store status information of NFs and handle requests for other NFs to find an NF that can be accessed.

NWDAF (180) can analyze data collected from a network (e.g., 5G network) to support network automation and provide analysis results. NWDAF (180) can collect/store/analyze information from a network. NWDAF (180) can collect information from OAM (190), NF constituting the network (e.g., AMF (120), SMF (130), UPF (140), PCF (160), NRF (170), NEF (175), MFAF (177), DCCF (185), ADRF (187), UDR (188), and/or UDM (189)), UE, or AF (150). The NWDAF (180) can provide the analysis results to an unspecified NF (e.g., AMF (120), SMF (130), UPF (140), PCF (160), NRF (170), NEF (175), MFAF (177), DCCF (185), ADRF (187), UDR (188), and/or UDM (189)), OAM (190), UE, or AF (150). The analysis results can be independently utilized by each NF (e.g., AMF (120), SMF (130), UPF (140), PCF (160), NRF (170), NEF (175), MFAF (177), DCCF (185), ADRF (187), UDR (188), and/or UDM (189)), OAM (190), UE, or AF (150).

FIG. 2 is a diagram for explaining a network data analysis process according to one embodiment.

NWDAF (210) can provide analytics information subscription service (Nnwdaf_AnalyticsSubscription service) to NF (230). NF (230) can be UE (100), RAN (110), AMF (120), SMF (130), UPF (140), AF (150), PCF (160), NRF (170), NEF (175), MFAF (177), DCCF (185), ADRF (187), and/or OAM (190) of FIG. 1. NF (230) can also be an NWDAF different from NWDAF (210).

The analysis information subscription service may enable subscription and unsubscription of network data analysis results generated by the NWDAF (210). In addition, the analysis information subscription service may enable selective subscription and unsubscription of analysis accuracy information. The analysis information subscription service may be segmented into a method of periodically receiving network analysis results and/or analysis accuracy information according to the needs of the network function of the NF (230) subscribing to the service, or receiving analysis results and/or analysis accuracy information when a specific condition is satisfied. The analysis information subscription service may be provided through three operations: subscribe, unsubscribe, and notify.

The subscription operation (Nnwdaf_AnlayticsSubscription_Subscribe operation) can subscribe to NWDAF analytics and/or analytics accuracy information using specific parameters. The subscription operation (Nnwdaf_AnlayticsSubscription_Subscribe operation) can optionally subscribe to analytics accuracy information. The subscription operation (Nnwdaf_AnlayticsSubscription_Subscribe operation) can include required input and/or optional input. The required input can include a single network slice selection assistance information (S-NSSAI), an event identifier or analytics identifier, a notification target address, and event reporting information. The optional input can include information additionally required for analytics information processing. For example, the optional input can include event filter or analytics filter information. Of course, the present invention is not limited to the above examples.

Examples of one or more parameters included in the Nnwdaf_AnlayticsSubscription_Subscribe service operation might include:

(1) Inputs, Required:

- Analysis ID (or set of analysis IDs);

- Target of Analytics Reporting;

- Notification Target Address (+Notification Correlation ID);

- Analytics Reporting Parameters (e.g. Analytics target period, etc.).

Targets for analysis reports can be provided by individual analysis ID.

(2) Inputs, Optional:

- Analytics Filter Information;

- Time window for collecting historical analytics;

- Subscription Correlation ID (e.g. if you modified an analytics subscription);

- preferred level of accuracy of the analytics;

- Preferred level of accuracy per analytics subset;

- Reporting Thresholds;

- maximum number of objects requested (maximum);

- Preferred order of results, maximum number of SUPIs requested (SUPImax);

- time when analytics information is needed;

- Analytics Metadata Request;

- NWDAF identifier (or set of NWDAF identifiers) used by a consumer NF (e.g., NWDAF service consumer) when aggregating multiple analytics subscriptions;

- Dataset Statistical Properties;

- Output strategy;

- Data time window;

- Serving area or NF ID of the consumer NF;

- Information from previous analytics subscriptions (e.g., NWDAF identifiers (e.g., instance ID or set ID), analytics ID(s) (e.g., SUPI and analytics filter information for UE-related analytics), and subscription correlation ID);

- Use case context (i.e., the environment and use case in which the analytics (or analytics information) is used).

- Analytics Accuracy Request information.

-Analytics feedback information

Indicates that the consumer NF of the analytics has taken action(s) that were influenced by previously provided analytics (e.g., analytics results), which may or may not have affected the ground truth data corresponding to the requested analytics ID at the time the prediction refers to, and consequently affects the monitoring of ML model accuracy by the subscription, using the following parameter(s):

- The corresponding analytics ID(s) used to take the action(s);

- (where possible) an indication of whether the action affects ground truth data;

-Time stamp(s) at which the action(s) were performed.

Analysis filter information, reporting thresholds, maximum number of requested entities (max), maximum number of requested SUPIs (SUPImax), analysis metadata requests, dataset statistical properties, output strategy, data time window, and time for which analysis information is required can be provided for each individual analysis ID.

Analytics feedback information may be included in modification requests to existing analytics subscriptions.

(3) Outputs Required:

If the subscription is accepted, the subscription correlation ID (required to manage this subscription)

If the subscription is not accepted, an error response

(4) Outputs, Optional:

If available and the Consumer NF (e.g., Analytics Consumer) requests an immediate report, the first applicable Analytics report may be included.

If the target of the analytics report is SUPI or GPSI, the subscription may not be accepted (i.e., an error is sent to the user if user consent is not granted). If the target of the analytics report is an Internal Group Id or a list of SUPI/GPSIs or a UE, no error is sent, but the SUPI or GPSI may be skipped if user consent is not granted.

In the case of an unsubscribe operation (Nnwdaf_AnlayticsSubscription_Unsubscribe operation), NF (230) transmits subscription identifier information to NWDAF (180), and NWDAF (210) can transmit a message indicating that unsubscribe has been confirmed as output to NF (230) that requested unsubscribe.

The notification operation (Nnwdaf_AnlayticsSubscription_Notify operation) may be for NWDAF (210) to notify NF (230) that has successfully subscribed to the analytics information subscription service of the specified network data analysis results and/or analytics accuracy information periodically or when a specific condition is satisfied. The notification operation may include an event identifier or analytics ID (or analytics information ID) and a notification target address.

NWDAF (210) can provide an analysis information request service (Nnwdaf_AnalyticsInfo service) to NF (230). Unlike an analysis information subscription service, the analysis information request service can mean a service in which NF (230) requests analysis and/or analysis accuracy information on specific information and receives the result value immediately upon completion of the request. The operation of the analysis information request service can consist of a request and a response. NF (230) requesting analysis information and/or analysis accuracy information can send an analysis information request message to NWDAF (180).

NWDAF (210) can transmit analysis results and/or analysis accuracy information to each NF (230) that requested the analysis results and/or analysis accuracy information. The analysis results and/or analysis accuracy information can be used to optimize the performance of operations (or network functions) performed by each NF (230) (e.g., QoS (Quality of Service) management, traffic control, mobility management, load distribution, and power management of terminals).

NF (230) may be a consumer NF (or demand NF) requesting analysis results and/or analysis accuracy information from NWDAF (210), and may provide feedback (e.g., analytics feedback information) on the analysis results to NWDAF (210). Each NF (230) may be a consumer NF (e.g., service consumer NF) of a network data analysis service. NWDAF (210) may perform a role of collecting and analyzing data from each NF (230) to generate analysis results and/or analysis accuracy information requested from consumer NFs, and may collect analysis feedback information from consumer NFs to which NWDAF (210) provided analysis to improve the accuracy of the analysis information. NWDAF (210) may transmit the analysis results and/or analysis accuracy information to the consumer NF that transmitted the analysis request (e.g., including analysis and/or analysis accuracy information). Accordingly, NWDAF (210) may be a provider NF of analysis results and/or analysis accuracy information requested by a consumer NF. NWDAF (210) may be a service provider NF of a service that provides analysis results requested by a consumer NF.

NWDAF (210) may include one or more of AnLF (Analytics logical function) and MTLF (Model Training logical function). NWDAF (210) may include each of MTLF and AnLF, or support both.

An NWDAF including AnLF (e.g., NWDAF (210)) can perform inference and derive analytics information and/or analytics accuracy information (e.g., derive statistics and/or predictions and/or analytics accuracy based on an analytics consumer request or a request from an analytics model provider (e.g., NWDAF including MTLF). An NWDAF including AnLF can expose analytics services for network data (e.g., Nnwdaf_AnalyticsSubscription or Nnwdaf_AnalyticsInfo).

An NWDAF (e.g., NWDAF (210)) including an MTLF can train a Machine Learning (ML) model and expose new training services (e.g., providing an untrained initial version or a trained model).

NWDAF including MTLF can register (e.g., register with NRF) ML model provisioning services, training services, and monitoring services (e.g., Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelUpdate, Nnwdaf_MLModelTraining, Nnwdaf_MLModelTrainingInfo) when ML model provisioning and/or training is possible for an Analytic ID.

An NWDAF that includes AnLF can register an ML model monitoring service (e.g., Nnwdaf_MLModelMonitor) when ML model usage and/or monitoring is possible (e.g., registering it with an NWDAF that includes MTLF).

NWDAF including MTLF can collect feedback on analysis results using ML models provided through ML model monitoring service (e.g., analytics feedback information) and/or analytics accuracy information from NWDAF including AnLF.

Below we will explain in detail how NWDAF determines the accuracy of the analysis information.

NWDAF has accuracy checking capability for analytics and/or ML models and can provide accuracy information to consumers upon request or use it for internal processes.

Input data is collected from the data producer NF when there is a request for inference/prediction by analysis ID of NWDAF for a specific time period in the future, and ground truth data can be collected from the data producer NF corresponding to the requested analysis ID at the time referenced by the prediction. The ground truth data can be actual measured data observed at the time referenced by the prediction.

If the analytics feedback information indicates that actions have been triggered by the consumer's analytics output, ground truth data may be affected.

Analytics/ML model accuracy monitoring can be performed by comparing predictions made using the currently trained ML model and its corresponding ground truth data (e.g., corresponding true observed events). Analytics/ML model accuracy information (e.g., the output of analytics/ML model accuracy monitoring) represents a general performance measure for each of the analytics and ML models, which can consist of the number of correct predictions out of the total predictions and the corresponding number of samples.

How MTLF/AnLF determines whether a prediction is correct may vary depending on the implementation.

An NWDAF with accuracy checking capability (e.g., an NWDAF that includes AnLF and/or MTLF) may decide to initiate analysis accuracy monitoring based on:

- A request from an analytics accuracy consumer. For example, an analytics accuracy consumer could be an NWDAF including AnLF and/or an NWDAF including MTLF and/or an analytics consumer NF.

- Analysis feedback information that can be provided by the analysis consumer NF.

AnLF with analysis accuracy check capability can provide or notify the analysis consumer of the service of the accuracy information of the analysis ID, and if the analysis accuracy does not meet the analysis consumer's requirements, the analysis consumer can stop using the analysis for a certain period of time or receive a new analysis. In addition, if an updated analysis for the provided analysis ID can be generated within the correction time period, the updated analysis can be provided upon the analysis consumer's request.

AnLF, which has the capability of analyzing accuracy, can determine the analysis accuracy information based on:

- Comparison of predictions generated based on ML models with corresponding ground truth data

- Determine analysis accuracy by comparing analysis accuracy using multiple ML models.

MTLF, which has the ability to check ML model accuracy, can determine ML model performance degradation based on:

- Data including input data and/or analysis results and/or ground truth data (e.g. ground truth data collected from various data sources NFs, DCCF, AnLF, ADRF or constructed by OAM); or

- AnLF providing notifications about analysis accuracy information; or

- AnLF, which provides analysis feedback information on the analysis generated by the ML model.

NWDAF with MTLF can inform ML model consumers of the resulting ML model accuracy degradation by re-selecting a new ML model or retraining an existing ML model. NWDAF with MTLF can also consider the class of untrustworthy AFs when used as data sources.

FIG. 3 is a diagram for explaining the operation of NWDAF according to one embodiment.

NWDAF (310) may include at least one of AnLF and MTLF, and NWDAF (330) may include MTLF.

NWDAF (310) can use provisioning service tasks (e.g., Nnwdaf_MLModelProvision) and training service tasks (e.g., Nnwdaf_MLModelTraining) for ML models trained in NWDAF (330).

AnLF can perform inference and derive analytics information (e.g., derive statistics and/or predictions based on analytics consumer requests) and expose analytics services (e.g., Nnwdaf_AnalyticsSubscription or Nnwdaf_AnalyticsInfo). MTLF can train ML models and expose new training services (e.g., serving trained ML models and training ML models). AnLF and/or MTLF can perform ML model analytics accuracy monitoring and expose analytics accuracy information of the ML models. The ML model analytics accuracy monitoring operation can include an operation of generating analytics accuracy information of the ML models.

AnLF can support data analysis information request service (e.g., Nnwdaf_AnalyticsInfo) or analytics subscription service (e.g., Nnwdaf_AnalyticsSubscription). MTLF can support services such as ML model provisioning service (e.g., Nnwdaf_MLModelProvision), ML model information request service (e.g., Nnwdaf_MLModelInfo), ML model training service (e.g., Nnwdaf_MLModelTraining), and ML model training information request service (e.g., Nnwdaf_MLModelTrainingInfo).

NWDAF (310) can subscribe/unsubscribe to ML model accuracy (e.g., analysis accuracy for ML models) information monitored via Nnwdaf_MLModelMonitor service. Nnwdaf_MLModelMonitor service can additionally provide analysis feedback information and/or analysis accuracy information for analysis generated by NWDAF (310). NWDAF (310) can register the usage and monitoring capability of ML models with NWDAF (330), which is a model provider. Nnwdaf_MLModelMonitor service can include Nnwdaf_MLModelMonitor_Subscribe service operation, Nnwdaf_MLModelMonitor_Unsubscribe service operation, Nnwdaf_MLModelMonitor_Notify service operation, Nnwdaf_MLModelMonitor_Register service operation, and Nnwdaf_MLModelMonitor_Deregister service operation.

(1) Nnwdaf_MLModelMonitor_Subscribe service operation

The Nnwdaf_MLModelMonitor_Subscribe service operation can subscribe to an NWDAF (e.g., NWDAF (330)) that provided an ML model for ML model accuracy information (e.g., analysis accuracy for the model) and analysis feedback information (e.g., analysis feedback information for an analysis generated from an NWDAF (e.g., NWDAF (310)) that includes AnLF) using certain parameters.

i) Inputs, Required:

Unique ML Model Identifier(s) (or a set of unique ML Model Identifiers), Notification Target Address (+Notification Correlation ID).

ii) Inputs, Optional:

A Subscription Correlation ID (e.g., in case of modifying an ML model monitor subscription), Accuracy metrics indicating the metrics to calculate accuracy information, ML model accuracy information period indicating the reporting periodicity for which information can be reported, and Accuracy reporting threshold indicating the reporting condition under which accuracy information must be reported.

iii) Outputs, Required:

If the subscription is accepted: subscription correlation ID (e.g. required to manage the subscription), Expiry time (e.g. required if the subscription can expire based on operator policy).

iv) Outputs, Optional: None.

(2) Nnwdaf_MLModelMonitor_Unsubscribe service operation

A consumer NF may unsubscribe from NWDAF for information on ML model accuracy (e.g., analysis accuracy for an ML model) and analysis feedback information on analyses generated by NWDAF.

i) Inputs, Required: Subscription correlation ID.

ii) Outputs, Required: Operation execution result indication.

iii) Outputs, Optional: None.

(3) Nnwdaf_MLModelMonitor_Notify service operation

NWDAF can notify consumer instances that subscribe to a specific NWDAF service about ML model accuracy (e.g., analysis accuracy for an ML model) information and analysis feedback information for analyses generated by NWDAF (same as NWDAF above).

i) Inputs, Required: Notification Correlation Information, one or more of the following:

- Tuple (e.g., unique ML model identifier, ML model accuracy information): ML model accuracy information may include a deviation value indicating the deviation of predictions generated using the ML model from the ground truth data, network data indicated by a DataSetTag with an ADRF ID when the deviation occurred (e.g., can be used by NWDAF including MTLF for possible ML model retraining), accuracy metrics requested in the subscription service operation; and

- Analysis Feedback Information: Indicates that a consumer NF of an analysis generated by a provisioned ML model has taken action(s) influenced by the analysis, including the following parameters:

- The corresponding analytics ID(s) used to take the action(s);

- The corresponding ML model identifier(s) used to generate the analysis;

- (if possible) an indication of whether the action affects ground truth data;

-Time stamp(s) at which the action(s) were performed.

- (If available) Use case context (i.e. the environment and use case in which the analytics (or insights) are used). ii) Inputs, Optional: Validity period.

iii) Outputs, Required: Operation execution result indication.

iv) Outputs, Optional: None.

(4) Nnwdaf_MLModelMonitor_Register service operation

Consumers can register their usage and monitoring capabilities for ML models in the NWDAF, which includes MTLF.

i) Inputs, Required: Consumer NF ID, unique ML model identifier.

ii) Inputs, Optional: Endpoint address of the Nnwdaf_MLModelMonitor_Subscribe service operation.

iii) Outputs, Required: ML model monitoring registration ID.

iv) Outputs, Optional: None.

(5) Nnwdaf_MLModelMonitor_Deregister service operation

If a consumer no longer uses or monitors the accuracy of analytics generated using an ML Model, the consumer may cancel the previous ML Model Monitor registration in the NWDAF, which includes the MTLF.

i) Inputs, Required: ML model monitoring registration ID.

ii) Inputs, Optional: None.

iii) Outputs, Required: None.

iv) Outputs, Optional: None.

FIG. 4 is a drawing for explaining the structure of NWDAF according to one embodiment.

Referring to I of FIG. 4, the operation of the NWDAF (410) including the MTLF will be described. The NWDAF (410) can receive an initial version of the ML model from the Model provisioning server (Operator) (403), the Model Provisioning server (3 ^rd Party) (405), or another NWDAF (407) including the MTLF. Then, the NWDAF (410) can train the initial version of the ML model, and then provide the trained ML model to the NWDAF (415) including the AnLF or the NWDAF (417) including the MTLF through the ML model provisioning service (e.g., Nnwdaf_MLModelProvision service) or the ML model information service (e.g., Nnwdaf_MLModelInfo service). In addition, for updating the ML model, the NWDAF (410) can use the Nnwdaf_MLModelTraining or Nnwdaf_MLModelTrainingInfo service.

Referring to II of FIG. 4, the operation of the NWDAF (430) including AnLF will be described. The NWDAF (430) can collect data from a DCCF device and/or a data source (e.g., NF or ADRF). The NWDAF (430) can receive an ML model from the NWDAF (435) including the MTLF. The NWDAF (430) can analyze the collected data using the ML model. The NWDAF (430) can provide the analysis results of the data to the consumer NF device (437) in the form of statistics or predictions.

FIG. 5 is a diagram illustrating an example of a procedure for analysis subscription/unsubscription according to one embodiment.

FIG. 5 may be intended to illustrate an analytics subscription/unsubscription (or unsubscription) procedure by an NWDAF service consumer (e.g., referred to as a consumer NF or service consumer NF).

An NWDAF service consumer may be an AMF (120), an SMF (130), a UPF (140), a PCF (160), an NRF (170), an NEF (175), an MFAF (177), an NWDAF (180), a DCCF (185), an ADRF (187), an OAM (190), a UE, or an AF (150) of FIG. 1. The NWDAF may be substantially the same as the NWDAF described with reference to FIGS. 1 to 4 (e.g., an NWDAF including one or more of AnLF and MTLF).

NWDAF service consumers can subscribe and/or unsubscribe from NWDAF to analytics information (e.g., notifications about analytics information). NWDAF service consumers can use the Nnwdaf_AnalyticsSubscription service to subscribe and/or unsubscribe from NWDAF to analytics information. Additionally, NWDAF service consumers can use the Nnwdaf_AnalyticsSubscription service to modify existing analytics subscription(s).

At operation 510, an NWDAF service consumer can subscribe or unsubscribe to analytics information by invoking the Nnwdaf_AnalyticsSubscribe_Subscribe/Nnwdaf_AnalyticsSubscribe_Unsubscribe service operations.

When a subscription for analytics information is received from a NWDAF service consumer, NWDAF (505) can determine whether new data collection needs to be triggered.

If the service invocation is for a subscription modification, the NWDAF service consumer may include an identifier to be modified (e.g., a Subscription Correlation ID) in the call to Nnwdaf_AnalyticsSubscription_Subscribe. Additionally, if the NWDAF service consumer has taken an action(s) that was affected by analytics information previously received in operation 530 (e.g., may or may not affect ground truth data corresponding to the requested Analytics ID at the time the prediction references it), the NWDAF service consumer may include analytics feedback information in the call to Nnwdaf_AnalyticsSubscription_Subscribe.

When the subscription involves outbound roaming users, the NWDAF (e.g., the NWDAF within the Home PLMN (HPLMN)) may decide to retrieve or subscribe to input data or analytics from the Visited PLMN (VPLMN).

If the subscription involves inbound roaming users, the NWDAF (e.g., within a VPLMN) may decide to retrieve or subscribe to input data or analytics from the HPLMN.

At operation 530, if the NWDAF service consumer subscribes to the analytics information, the NWDAF can notify the NWDAF service consumer of the analytics information by calling the Nnwdaf_AnalyticsSubscription_Notify service operation based on the request of the NWDAF service consumer (e.g., Analytics Reporting Parameters). If the NWDAF provides a Termination Request, the NWDAF service consumer can unsubscribe from the analytics information by calling the Nnwdaf_AnalyticsSubscription_Unsubscribe service operation.

When calculating analytics/ML model accuracy information with retrieved analytics feedback information, in addition to comparing the prediction of the ML model to the corresponding ground truth data, NWDAF may additionally determine/consider whether the action(s) taken by the NWDAF service consumer impacted the ground truth data corresponding to the requested analytics ID at the time the prediction references. This may impact ML model accuracy monitoring.

FIG. 6 is a diagram illustrating another example of a procedure for analysis subscription/unsubscription according to one embodiment.

FIG. 6 is intended to illustrate an analysis subscription/unsubscription procedure of AF(s) via NEF, and may be intended to illustrate an interaction between an AF performed via NEF and an NWDAF. The NWDAF may be substantially identical to the NWDAF described with reference to FIGS. 1 to 5 (e.g., an NWDAF including one or more of AnLF and MTLF).

At operation 610, the NEF may control analytics exposure mapping between AF identifiers with allowed analytics IDs and/or associated inbound restrictions (e.g., those that apply to subscriptions to analytics IDs for AFs) and/or outbound restrictions (e.g., those that apply to notifications to analytics IDs for AFs).

AF can be configured with appropriate NEFs to subscribe to analytics information, allowed analytics ID(s), and allowed inbound restrictions (e.g., parameters and/or parameter values) for subscriptions for each analytics ID.

At operation 620, the AF can subscribe to or unsubscribe from analytics information via NEF by calling the Nnef_AnalyticsExposure_Subscribe/Nnef_AnalyticsExposure_Unsubscribe service operations. If the AF wishes to modify an existing analytics subscription in NEF, the AF may include an identifier to modify (e.g., a subscription correlation ID) in the Nnef_AnalyticsExposure_Subscribe call. Additionally, if the AF has taken any action(s) that are affected by analytics information previously received at operation 650 (e.g., which may or may not affect the ground truth data corresponding to the analytics ID requested at the time the prediction references), the AF may include analytics feedback information in the call to Nnef_AnalyticsExposure_Subscribe. If the analytics information subscription is approved by the NEF, the NEF may perform the following actions:

At operation 630, based on a request from .AF, NEF can subscribe, modify, or cancel a subscription to analytics information by calling Nnwdaf_AnalyticsSubscription_Subscribe/Nnwdaf_AnalyticsSubscription_Unsubscribe service operations. At operation 650, if NEF receives analytics feedback information, it can include the analytics feedback information in the call to Nnwdaf_AnalyticsSubscription_Subscribe.

If the parameters and/or parameter values of the AF request comply with the inbound restrictions in the analytics exposure mapping, NEF may forward the analytics ID, parameters and/or parameter values from the AF request in the subscription to the NWDAF service.

If a request from AF does not comply with the restrictions in the analytics exposure mapping, NEF may apply restrictions to the subscription request to NWDAF based on operator configuration (e.g. restrictions on parameters or parameter values of the Nnwdaf_AnalyticsSubscription_Subscribe service operation) or parameter mapping (e.g. geo coordinate mapping to TA(s)/cell-ID(s)).

NEF can record the association between an analysis request from AF and an analysis request sent to NWDAF.

NEF can use the NWDAF discovery procedure to select an NWDAF that supports the analysis information requested by the AF.

If the subscription involves outbound roaming users, the NWDAF within the HPLMN may decide to retrieve or subscribe to input data or analytics from the VPLMN.

If the AF request is a request to modify an existing analytics subscription, NEF can modify the analytics subscription identified by the identifier associated with the AF (e.g., subscription correlation ID) by calling Nnwdaf_AnalyticsSubscription_Subscribe.

At operation 640, if NEF has subscribed to analytics information, NWDAF can call the Nnwdaf_AnalyticsSubscription_Notify service operation to notify NEF of analytics information or a termination request.

At operation 650, when NEF receives a notification from NWDAF, NEF may send a notification to AF with analytics information or a Termination Request by calling Nnef_AnalyticsExposure_Notify service operation. NEF may apply outbound restrictions (e.g., restrictions on parameters or parameter values of Nnef_AnalyticsExposure_Notify service operation) to notifications to AFs based on analytics exposure mapping, and may apply parameter mapping for external usage (e.g., mapping TA, cell-id to geo coordinate). AF may check whether there is a Termination Request in Nnef_AnalyticsExposure_Notify.

When calculating analytics/ML model accuracy information with retrieved analytics feedback information, in addition to comparing the predictions of the ML model to the corresponding ground truth data, NWDAF may additionally determine/consider whether the action(s) taken by the AF(s) impacts the ground truth data corresponding to the requested analytics ID at the time the prediction refers to, or may begin evaluating the AF(s) used as data sources. This may impact ML model accuracy monitoring.

The Nnef_AnlayticsExposure_ service may include the Nnef_AnlayticsExposure_Subscribe service action, the Nnef_AnalyticsExposure_Unsubscribe service action, and the Nnef_AnalyticsExposure_Notify service action.

(1) Nnef_AnlayticsExposure_Subscribe service operation

NF consumers can subscribe to or modify existing subscriptions to analytics information.

i) Inputs, Required:

- Analysis ID (or set of analysis IDs);

- Analysis filter information

- Target of Analytics Reporting (e.g. UEs (e.g. GPSI), External Group Identifier, any UE),

- Analytic Reporting Information

- Notification Target Address (+Notification Correlation ID)

ii) Inputs, Optional:

- Subscription Correlation ID (e.g. if you modified an analytics subscription)

- Expiry time

-slice specific information

-Geographical area

- Analytics feedback information:

Indicates that the consumer NF of the analytics has taken action(s) that were influenced by previously provided analytics (e.g., analytics results), which may or may not have affected the ground truth data corresponding to the requested analytics ID at the time the prediction refers to, and consequently affects the monitoring of ML model accuracy by the subscription, using the following parameter(s):

- The corresponding analytics ID(s) used to take the action(s);

- (where possible) an indication of whether the action affects ground truth data;

- The timestamp(s) at which the action(s) were performed.

- (Optional) Use case context (i.e., the environment and use case in which the analytics (or analytics information) is used)

Analytics feedback information may only be included in modification requests to existing analytics subscriptions.

iii) Outputs Required:

If the subscription is accepted: subscription correlation ID, expiration time (required if the subscription can expire according to the operator's policy)

iv) Outputs, Optional: Includes the first applicable analysis report, if available.

(2) Nnef_AnalyticsExposure_Unsubscribe service operation

NF consumers can cancel their existing subscriptions to analytics information.

i) Inputs, Required: Subscription correlation ID

ii) Outputs Required: Operation execution result indication

(3) Nnef_AnalyticsExposure_Notify service operation

NEF may report analytics (e.g., analysis results) to previously subscribed NF consumers.

i) Inputs, Required: Analysis ID, alert correlation information, analysis information (e.g. defined by analysis ID).

ii) Inputs, Optional: Timestamp of analytics generation, Probability assertion, Termination Request

iii) Outputs Required: Operation execution result indication

Figure 7 illustrates an example of a ML model accuracy monitoring procedure according to one embodiment.

The NWDAF including MTLF can perform an MTLF-based ML model accuracy monitoring procedure. FIG. 7 may be for explaining an MTLF-based ML model accuracy monitoring procedure. The MTLF-based ML model accuracy monitoring procedure may be a procedure for judging ML model degradation (e.g., ML model performance degradation) based on newly collected test data and retraining or reprovisioning an existing ML model.

Fig. 7 can illustrate a procedure for monitoring the accuracy of a provisioned ML model using newly collected data. An NWDAF including AnLF can provide inference data to an NWDAF including MTLF for model accuracy monitoring, and an NWDAF including MTLF can determine retraining or reprovisioning of the ML model.

At operation 705, an analytics consumer (e.g., referred to as a consumer NF or a service consumer NF or an NWDAF service consumer) may initiate a subscription to an analytics exposure service for an NWDAF that includes AnLF.

In operation 710, the NWDAF containing the AnLF may request an ML model from the NWDAF containing the MTLF (e.g., an appropriate NWDAF) using the Nnwdaf_MLModelProvision_Subscribe service operation. The NWDAF containing the AnLF may include an analytics accuracy threshold that is used as an indicator (e.g., a monitoring indicator) for executing the accuracy monitoring operation. The NWDAF containing the AnLF may include a DataSetTag and/or an ADRF ID. These (e.g., the DataSetTag and/or the ADRF ID) may be used to retrieve and store inference data (e.g., input data, predictions, and ground truth data at the time the predictions refer to) relevant to the accuracy monitoring and retraining/reprovisioning of the ML model from the ADRF. The Nnwdaf_MLModelProvision_Subscribe may include the monitoring indicator, the DataSetTag, and/or the ADRF ID.

In operation 715, the NWDAF including the MTLF can provide the trained ML model to the NWDAF including the AnLF. The NWDAF including the MTLF can include accuracy information used to indicate the accuracy of the ML model during training. The Nnwdaf_MLModelProvision_Notify can include accuracy information.

In operation 720, the NWDAF including the AnLF can register the use of the ML model with the NWDAF including the MTLF. The NWDAF including the AnLF can indicate the capability to transmit analytics feedback information of an analytics consumer and/or ML model accuracy information about the ML model by registering the use of the ML model with the NWDAF including the MTLF.

At operation 725, due to the registration in operation 720, the NWDAF including the MTLF may subscribe to the NWDAF including the AnLF to obtain analytics feedback information from the analytics consumer and/or ML model accuracy information for the provisioned ML model by calling the Nnwdaf_MLModelMonitor_Subscribe service operation. This may be the case if the service operation (e.g., the Nnwdaf_MLModelMonitor_Subscribe service operation) is supported by the NWDAF including the AnLF.

At operation 730, the analytics consumer can send analytics feedback information with the Nnwdaf_AnalyticsSubscription_Subscribe message.

At operation 735, the NWDAF including the AnLF can call the Nnwdaf_MLModelMonitor_Notify service operation as requested at operation 725 to transmit the ML model accuracy information for the provisioned ML model and/or the analytics feedback information received from the analytics consumer. If the NWDAF including the MTLF receives the analytics feedback information or the ML model accuracy information, the NWDAF including the MTLF can trigger operations 740a through 790 to improve the ML model accuracy.

In steps 740a through 740f, the NWDAF including the MTLF may determine whether to collect new data from various data sources based on a request from the NWDAF including the AnLF (e.g., more than one) or the local policy to perform ML model accuracy monitoring and ML model retraining/reprovisioning:

In actions 740a and 740b, the NWDAF including the MTLF can collect new data for ML model accuracy monitoring, retraining, and reprovisioning from data source NFs and DCCF by calling Nnf_EventExposure_Subscribe and Ndccf_DataManagement_Susbscribe service actions, respectively (respectively).

In actions 740c and 740d, given an ADRF ID and/or a DataSetTag in action 710, the NWDAF containing the MTLF may retrieve historical data (e.g., historical analytics) from the ADRF specified by the NWDAF containing the AnLF in action 710 by calling the Nadrf_DataManagementRetrievalRequest or Nadrf_DataManagementRetrieval_Subscribe service actions. Otherwise, the NWDAF containing the MTLF may retrieve historical data (e.g., historical analytics) from the NWDAF containing the AnLF or the DCCF by calling the Ndccf_DataManagement_Subscribe or Nnwdaf_DataManagement_Subscribe service actions, respectively.

If the NWDAF containing AnLF does not include a DataSetTag with an ADRF ID in operation 710, the NWDAF containing MLTF may request a subscription to ADRF for analysis and data collection corresponding to the analysis generated by the ML model provided in operation 715.

At operation 740e, the NWDAF including the MTLF can subscribe to the UDM to be notified of changes in the subscription data for the target of the ML model report by calling the Nudm_SDM_Subscribe service operation, and the UDM can subscribe to be notified of modifications to the UE subscription data by calling the Nudr_DM_Subscribe service operation of the UDR.

At operation 740f, the NWDAF including the MTLF may collect fault prediction analytics data from the MDAS to determine the status of data source NFs using MDA requests, so that data quality can be considered for accuracy monitoring.

If an NWDAF containing MTLF has already collected new test data and performed ML model accuracy monitoring and retraining (e.g., triggered by another NWDAF containing AnLF for ML model accuracy monitoring and retraining), the NWDAF containing MTLF can decide whether to use the data for subscription based on its internal logic.

In operations 750a through 750f, the NWDAF including the MTLF can receive requested data from various sources as requested in operations 740a through 740f.

At operation 760, based on the analysis and data collected at operations 750a to 750f, the NWDAF including the MTLF can compute an accuracy using the prediction and actual measured data observed at the time the prediction refers to. The NWDAF including the MTLF can discard data of the data source if it is detected that the data quality of the data source is poor. When only input data and ground truth data are available, the NWDAF including the MTLF can generate a prediction with the collected input data to compute the accuracy.

How NWDAF, including MTLF, determines whether data from a data source is of good quality or should be discarded is up to the NWDAF implementation and configuration.

At operation 770, the NWDAF containing the MTLF can send an accuracy report (e.g., including the accuracy computed at operation 760) to the NWDAF containing the AnLF. For example, if a reporting threshold is met, the NWDAF containing the MTLF can call the Nnwdaf_MLModelProvision_Notify service operation to send an accuracy report to the NWDAF containing the AnLF.

At operation 780, Based on the computed accuracy, the NWDAF including MTLF can retrain/reprovision the ML model.

In operation 790, when the newly generated ML model or the retrained ML model is ready, the NWDAF including the MTLF can call the Nnwdaf_MLModelProvision_Notify service operation to send the newly generated ML model or the retrained ML model to the NWDAF including the AnLF. The NWDAF including the MTLF can send an accuracy report of the newly generated ML model or the retrained ML model to the NWDAF including the AnLF.

FIG. 8 illustrates a schematic block diagram of a device performing NWDAF according to one embodiment.

Referring to FIG. 8, according to one embodiment, a device (1000) (e.g., a server device) performing NWDAF may be substantially the same as the NWDAF (e.g., the NWDAF including AnLF or the NWDAF including MTLF) described with reference to FIGS. 1 to 7. The device (1000) may include a memory (1010) and a processor (1030). The device (1000) may operate with the NWDAF including AnLF or the NWDAF including MTLF.

The memory (1010) may store instructions (e.g., programs) executable by the processor (1030). For example, the instructions may include instructions for executing operations of the processor (1030) and/or operations of each component of the processor (1030).

The memory (1010) may be implemented as a volatile memory device or a nonvolatile memory device. The volatile memory device may be implemented as a DRAM (dynamic random access memory), a SRAM (static random access memory), a T-RAM (thyristor RAM), a Z-RAM (zero capacitor RAM), or a TTRAM (Twin Transistor RAM). The nonvolatile memory device can be implemented as an Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, Magnetic RAM (MRAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), Ferroelectric RAM (FeRAM), Phase change RAM (PRAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, and/or Insulator Resistance Change Memory.

The processor (1030) can execute computer-readable code (e.g., software) stored in the memory (1010) and instructions generated by the processor (1030). The processor (1030) can be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. The desired operations can include, for example, code or instructions included in a program. The hardware-implemented data processing device can include, for example, a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array).

The operation performed by the processor (1030) is substantially the same as the operation of the NWDAF (e.g., the NWDAF including AnLF or the NWDAF including MTLF) described with reference to FIGS. 1 to 7. Therefore, a detailed description is omitted.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the OS. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the described embodiments. For example, even if the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

Claims (16)

In terms of how a provider that provides an ML model operates,
The act of providing the first ML model to the consumer;
An operation of collecting data from data sources and evaluating the first model; and
An action of providing a second ML model to the consumer based on the evaluation results of the first model.
A method comprising:
In the first paragraph,
The above provider,
NWDAF (network data analytics function) including MTLF (Model Training logical function),
The above consumer,
A method, which is an NWDAF including AnLF (Analytics logical function).
In the first paragraph,
An action to register the use of the above consumer's ML model.
Including more,
The above registration is,
A method that represents the capability to transmit at least one of analysis feedback information for an analysis generated by the ML model and information on the analysis accuracy of the ML model.
In the first paragraph,
The above evaluation actions are:
An operation to monitor the accuracy of the first ML model by collecting data from data sources NF and DCCF.
A method comprising:
In the first paragraph,
The above evaluation actions are:
An operation of monitoring the accuracy of the first ML model by collecting ADRF data using at least one of ADRF ID and DataSetTag.
A method comprising:
In the first paragraph,
The above evaluation actions are,
An operation of collecting fault prediction analytics data from MDAS to monitor the accuracy of the first ML model.
A method comprising:
In the first paragraph,
The above evaluation actions are:
An action to subscribe to UDM to be notified of changes in subscription data for the target of the first ML model by calling the Nudm_SDM_Subscribe service action.
Including,
The above UDM subscribes to the UDR to receive notifications about UE subscription data by calling the Nudr_DM_Subscribe service operation.
In the first paragraph,
The above second ML model is,
A method, wherein the first ML model or a newly generated ML model is retrained based on data collected from the above data sources.
For a server device that provides an ML model,
processor; and
A memory electrically connected to the processor and storing instructions executable by the processor
Including,
When the instructions are executed by the processor, the processor performs a plurality of operations,
The above multiple actions are:
The act of providing the first ML model to the consumer;
An operation of collecting data from data sources and evaluating the first model; and
An action of providing a second ML model to the consumer based on the evaluation results of the first model.
A server device, including:
In Article 9,
The above server device,
NWDAF (network data analytics function) including MTLF (Model Training logical function),
The above consumer,
A server device, which is an NWDAF including AnLF (Analytics logical function).
In Article 9,
The above multiple actions are:
An action to register the use of the above consumer's ML model.
Including more,
The above registration is,
A server device having the capability to transmit at least one of analysis feedback information for analysis generated by the ML model and information on analysis accuracy of the ML model.
In Article 9,
The above evaluation actions are,
An operation to monitor the accuracy of the first ML model by collecting data from data sources NF and DCCF.
A server device, including:
In Article 9,
The above evaluation actions are,
An operation of monitoring the accuracy of the first ML model by collecting ADRF data using at least one of ADRF ID and DataSetTag.
A server device, including:
In Article 9,
The above evaluation actions are,
An operation of collecting fault prediction analytics data from MDAS to monitor the accuracy of the first ML model.
A server device, including:
In Article 9,
The above evaluation actions are,
An action to subscribe to UDM to be notified of changes in subscription data for the target of the first ML model by calling the Nudm_SDM_Subscribe service action.
Including,
The above UDM is a server device that subscribes to UDR to receive notifications about UE subscription data by calling the Nudr_DM_Subscribe service operation.
In Article 9,
The above second ML model is,
A server device, which is a first ML model or a newly generated ML model retrained based on data collected from the above data sources.

Images