WO2025214801A1

WO2025214801A1 - Method for monitoring accuracy and improving synthetic data generation

Info

Publication number: WO2025214801A1
Application number: PCT/EP2025/058730
Authority: WO
Inventors: Marvin Manalastas; Afsaneh GHAROUNI; Ece GOSHI; Colin Kahn
Original assignee: Nokia Technologies Oy
Current assignee: Nokia Technologies Oy
Priority date: 2024-04-10
Filing date: 2025-03-31
Publication date: 2025-10-16
Anticipated expiration: 2026-10-10
Also published as: GB2640273A

Abstract

A method includes receiving a request for analytics from a network function service consumer, obtaining real data from one or more data sources of the communication network, and determining, by the network data analytics function, a first accuracy of analytics predicted by a first trained machine learning model, wherein the first trained machine learning model is trained using a first training dataset comprising real data. The method further includes obtaining synthetic data generated by one or more synthetic data generator from a synthetic data controller, determining, by the network data analytics function, a second accuracy of analytics predicted by a second trained machine learning model, wherein the second trained machine learning model is trained using a second training dataset comprising the real data and synthetic data; comparing, by the network data analytics function, the first accuracy to the second accuracy and determining which of the first machine learning model and the second machine learning model predicts more accurate analytics based on the comparison.

Description

METHOD FOR MONITORING ACCURACY AND IMPROVING SYNTHETIC DATA GENERATION

RELATED APPLICATIONS

[0001] This patent application claims the benefit of priority of United Kingdom Patent Application No. 2405079.1 filed on April 10, 2024, which is hereby incorporated by reference as if reproduced in its entirety.

FIELD

[0002] Various example embodiments relate generally to wireless networks and, more particularly, to monitoring accuracy and improving synthetic data generation in wireless networks.

BACKGROUND

[0003] A third-generation partnership project (3 GPP) network may include a network function that performs analytics and shares analytics information generated with other network functions of the 3GPP network. This analytics information may be used to aid other network functions of the 3GPP network in making decisions regarding actions to be taken. For example, analytics information may be utilized to assist a network function in decisions such as handover.

[0004] Within a 3GPP network, various function may be provided analytics information and may utilize instances of artificial intelligence/machine learning (AI/ML) model(s) that may be trained with appropriate data. Once trained, an AI/ML model may then generate analytics information, such as statistical information and/or predictions based on new data input into the AI/ML model that may result in improved performance of various operational aspects of the 3 GPP network.

SUMMARY

[0005] In a first aspect of the present disclosure, there is provided a method that includes receiving a request for analytics from a network function service consumer, obtaining real data from one or more data sources of the communication network, and determining, by the network data analytics function, a first accuracy of analytics predicted by a first trained machine learning model, wherein the first trained machine learning model is trained using a first training dataset comprising real data. The method further includes obtaining synthetic data generated by one or more synthetic data generator from a synthetic data controller, determining, by the network data analytics function, a second accuracy of analytics predicted by a second trained machine learning model, wherein the second trained machine learning model is trained using a second training dataset comprising the real data and synthetic data, comparing, by the network data analytics function, the first accuracy to the second accuracy and determining which of the first machine learning model and the second machine learning model predicts more accurate analytics based on the comparison.

[0006] In some or all examples of the first aspect, the comparing comprises determining that the first accuracy is less than the second accuracy, and sending a response to the request for analytics, the response comprising the analytics predicted by the second machine learning model.

[0007] In some or all examples of the first aspect, the comparing comprises determining that the first accuracy is greater than the second accuracy, and sending a response to the request for analytics, the response comprising the analytics predicted by the first machine learning model.

[0008] In some or all examples of the first aspect, the method further comprises sending, to the synthetic data controller, information indicative of which of the first machine learning model and the second machine learning model predicts more accurate analytics.

[0009] In some or all examples of the first aspect, the determining of the second accuracy comprises determining a third accuracy of analytics predicted using a third machine learning model trained using a training dataset comprising the real data and a first percentage of the synthetic data, determining a fourth accuracy of analytics predicted using a fourth machine learning model trained using a training dataset comprising the real data and a second percentage of the synthetic data, comparing the third accuracy to the fourth accuracy, determining the second accuracy to be third accuracy when the third accuracy is greater than the fourth accuracy, and determining the second accuracy to be the fourth accuracy when the fourth accuracy is greater than the third accuracy.

[00010] In the preceding example of the first aspect, the method further comprises sending to the synthetic data controller information relating to the third accuracy and the fourth accuracy.

[00011] In some or all examples of the first aspect, the one or more data sources comprise an access and mobility management function (AMF), application functions (AFs), or operations and management function (0AM) of the communication network.

[00012] In a second aspect of the present disclosure, there is provided a method that includes receiving, by a synthetic data generator from a network data analytics function, information relating to a comparison of a first accuracy for analytics predicted using a first machine learning model trained using real data to a second accuracy for analytic predicted using a second trained machine learning model trained using a combination of real data and synthetic data; and based upon the information, determining by a synthetic data generator, whether to store or discard the synthetic data.

[00013] In some or all examples of the second aspect, the real data used to train the first machine learning model is from one or more of an access and mobility management function (AMF), application functions (AFs), or operations and management function (0AM).

[00014] In some or all examples of the second aspect, the synthetic data used to train the second machine learning model is from a synthetic data source.

[00015] In some or all examples of the second aspect, the determining comprises determining that the second accuracy is greater than the first accuracy and causing the synthetic data to be stored in a data repository.

[00016] In some or all examples of the second aspect, the determining comprises determining that the first accuracy is greater than or equal to the second accuracy, discarding the synthetic data.

[00017] In some or all examples of the second aspect, the determining of the second accuracy includes comparing a third accuracy including a use of a first percentage of the second data as the training data with a fourth accuracy including the use of a second percentage of the second data as the training data.

[00018] In the preceding example of the second aspect, the method further includes sending information relating to the third accuracy and the fourth accuracy.

[00019] In a third aspect of the present disclosure, an apparatus includes at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to perform the method of any of the foregoing examples of the first and second aspects.

[00020] In a fourth aspect of the present disclosure, a non-transitory computer-readable medium storing instructions which, when executed by at least one processor of an apparatus, cause the apparatus at least to perform the method of any of the foregoing examples of the first and second aspects.

[00021] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are defined in the dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[00022] Some example embodiments will now be described with reference to the accompanying drawings. [00023] FIG. l is a diagram of an example embodiment of a wireless network, according to one illustrated aspect of the disclosure;

[00024] FIGS. 2A-2D are diagrams of example systems for synthetic data generation, according to one illustrated aspect of the disclosure;

[00025] FIGS. 3A-3B are diagrams of example systems for a synthetic data repository (SDR), according to one illustrated aspect of the disclosure;

[00026] FIG. 4 is a diagram of an example embodiment of signals and operations among an analytics service consumer, a NWDAF, real data sources, a synthetic data controller, and synthetic data generators, according to one illustrated aspect of the disclosure; and

[00027] FIG. 5 is a diagram of an example embodiment of components of an apparatus for a core network of a wireless network, according to one illustrated aspect of the present disclosure.

DETAILED DESCRIPTION

[00028] In the following description, certain specific details are set forth in order to provide a thorough understanding of disclosed aspects. However, one skilled in the relevant art will recognize that aspects may be practiced without one or more of these specific details or with other methods, components, materials, etc. In other instances, well-known structures associated with transmitters, receivers, or transceivers have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the aspects.

[00029] Reference throughout this specification to “one aspect” or “an aspect” means that a particular feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. Thus, the appearances of the phrases “in one aspect” or “in an aspect” in various places throughout this specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[00030] Embodiments described in the present disclosure may be implemented in wireless networks, such as, without limitation, 5G networks, 5G Advance networks, 6G networks, and any other future wireless networks that operate in accordance with future radio access technologies, such as 3 GPP or European Telecommunications Standards Institute (ETSI) standards.

[00031] 5G networks include a network data analytics function (NWDAF) that is configured to generate and provide analytics, such as statistics and predictions. For example, the NWDAF may generate analytics using trained artificial intelligence/machine learning (AI/ML) models and real data that may be used by other network functions of the 5G network or an operations and management (0AM) entity of the 5G network. The NWDAF may also train AI/ML models that generate analytics. The AI/ML models require data for generating analytics and/or training an AI/ML model that generates analytics (generally referred as AI/ML model training). For example, availability of data for generating analytics, and/or artificial intelligence/machine learning (AI/ML) model training may affect an accuracy of analytics generated by the NWDAF or an accuracy of a trained AI/ML model that generates the analytics (e.g., an accuracy of an AI/ML model that generates analytics after training is of the AI/ML model completed). Availability of data may also affect an amount of time (e.g., latency) a NWDAF takes to generate analytics (e.g., generate predictions and/or statistics), and/or train an AI/ML model that generates analytics. In various embodiments, the NWDAF may obtain real data that the NWDAF uses to generate analytics which may be used by other network functions and/or an 0AM entity of a 5G network to improve the performance of the 5G network. Real data may comprise data that is collected by Network Functions and provided by to the NWDAF (e.g., data related to events that observed by the network function of a 5G network (e.g., AMF, SMF, PCF, etc.)) and data collected by 0AM entity and provided to the NWDAF. Examples of data collected by a NF may comprise, for example, data related to a load of the NF (generally referred to as NF load of a NF), data related to a status of a NF (generally referred to as a NF status of a NF), data related to resource usage of the NF (generally referred to as NF resource usage). Examples of data collected by an 0AM entity may comprises data indicative of a throughput of data provided to a UE, data indicative of a delay in transporting data through the 5G network, data indicative of traffic volume in a cell of the 5G network, data indicative of traffic volume in a network slice of the 5G network, data indicative of a subscribed QoS, data indicative of a measured QoS, data indicative of locations of UE’s that are communicating with the 5G network, data indicative of a application identities, and/or data indicative of locations of applications.

[00032] As used herein, the terms “transmit to,” “receive from,” and “cooperate with,” (and their variations) include communications that may or may not involve communications through one or more intermediate devices or nodes. The term “acquire” (and its variations) includes acquiring in the first instance or reacquiring after the first instance. The term “connection” may mean a physical connection or a logical connection.

[00033] FIG. 1 shows a schematic representation of a communication network 100 (e.g., a 5G network) that a user equipment (UE) 102 may access to communicate with application servers (not shown) hosting third party application functions (not shown) via data network 104. The communication network 100 comprises a radio access network 106 (e.g., a NG- RAN) and a core network 108 (e.g., a 5G core network (5GC)) that operate based on the 5th generation radio access technology described in the 3rd Generation Partnership Project (3GPP) standard for new radio. The core network 108 is connected to an operations and maintenance (0AM) entity 110 of the communication network 100 as described in further detail below.

[00034] The radio access network 106 comprises one or more radio access network (RAN) nodes (otherwise referred to as base stations), such as a gNodeB (gNB). A radio access network node may comprise a central unit (e.g., gNB-CU) and one or more distributed units (e.g., one or more gNB-DUs) linked to the central unit (e.g., gNB-CU) by a Fl interface. The central unit (e.g., gNB-CU) may comprise a control plane (CP) unit (e.g., gNB-CU-CP) and one or more user plane (UP) units (e.g., gNB-CU-UP) linked to the CP unit (e.g., gNB-CU-CP) by an El interface.

[00035] The core network 108 (e.g., a 5GC) has a service-based architecture and comprises a plurality of network functions, including an access and mobility management function (AMF), an authentication server function (AUSF), a network exposure function (NEF), a network repository function (NRF), a network slice selection function (NSSF), a policy control function (PCF), a session management function (SMF), a user plane function (UPF), a united data repository (UDM) and a network data analytics function (NWDAF). Other network functions of the core network 108, such as a binding support function (BSF), a charging function (CHF), etc. are not included for ease of illustration. The functionalities of the network functions of the core network are well known to a person skilled in the art and hence are not described in detail.

[00036] Each network function (NF) of the core network 108 may provide one or more services to other network functions of the core network 108 via Application Programming Interfaces (APIs). Each NF can also register itself and the services it supports (e.g., the services it offers other network functions) to the NRF of the core network 108. The NRF may be used by any authorized NF to discover other NFs (or instances of NFs) and the services the other NFs support (e.g., services the other NFs provide). Any authorized NF may consume (e.g., use) the services provided and exposed by another NF. A NF that consumes a service of another NF is generally referred to as a NF service consumer. A NF that provides and exposes one or more of its services is referred to as a NF service producer. [00037] As used herein, the term “network apparatus” may refer to any computing device or computing system (e.g., cloud computing system) of the network, such as a server. The present disclosure describes embodiments related to 5G networks defined by 3rd Generation Partnership Project (3GPP). However, it is contemplated that embodiments relating to other networks, such as 6^th generation radio networks, are encompassed within the scope of the present disclosure. For example, although additional detail is provided below, in various embodiments, a network apparatus may implement or include a synthetic data controller (SDC), and/or a synthetic data generator (SDG), or other network entities or network functions of a core network described below.

[00038] FIG. 1 provides an example and is merely illustrative of a communication network 100. Persons skilled in the art will understand that the communication network 100 includes components not illustrated in FIG. 1 and will understand that other user equipment (e.g. other UEs) may be in communication in the communication network 100 (hereinafter referred to as network 100). In various embodiments various components and combinations of components shown in FIG. 1 may be included in the network 100.

[00039] As described above, availability of data for the NWDAF to generate analytics, and/or train one or more AI/ML models that generate analytics in a 3GPP network may limit the accuracy of analytics that are generated by the NWDAF and the amount of time (e.g., latency) the NWDAF takes to generate analytics (e.g., generate statistical information (e.g., statistics) related to past events (e.g., mobility events for UEs) or predictive information indicative of future event) based on a service request received from an analytics consumer of an analytics service provided by NWDAF and/or perform AI/ML model training based on a service request received from a consumer of model training service provided by the NWDAF. In various embodiments, the NWDAF may receive data (e.g., real data) which can be used by the NWDAF to generate analytics which can be used by an analytics consumer to improve the performance of a network (e.g., communication network 100).

[00040] In some embodiments, sufficient real data to generate analytics or train an AI/ML model that generates analytics may not be readily available or may not be collected in time. That is, for example, a sufficient amount of real data that is needed to generate analytics and/or train an AI/ML which satisfies requirements for the collection of real data and/or requirement for analytics generated using real data (e.g., requirements related to sampling ratio of real data, a maximum latency for collecting the real data, a maximum latency for generating analytics based on the real data, an accuracy of analytics that are to be generated based on the real data and/or an accuracy of an trained AI/ML model that is the result of AI/ML model training using real data) may not be collected or readily available. Accordingly, synthetic data may be used to supplement real data to generate analytics and/or train an AI/ML model.

[00041] However, using synthetic data to address the limitations of real data being scarce may introduce additional considerations. For example, not all synthetic data (e.g., synthetically generated data) has good quality and not all synthetic data are useful (e.g., improves the accuracy of analytics generated by a NWDAF and/or improves the training of AI/ML models that are used to generate analytics).

[00042] Synthetic data quality can be measured using metrics such as similarities in statistical properties with the real data (e.g., mean, median, standard deviation, unique and missing values, range (minimum value, maximum value), etc.), differences in the correlation between real data and synthetic data should be similar or close to each other), resemblance of the data distribution of real data and the data distribution of synthetic data (should resemble one another).

[00043] One example to assess the usefulness of using synthetic data when training a AI/ML model can be based on the disparity in performance of a first trained AI/ML (e.g., a first AI/ML model trained using a dataset comprising synthetic data and historical real data) and a second trained AI/ML model (e.g., a second AI/ML model trained using a dataset that does not include synthetic data (e.g., a dataset that includes only historical real data) at inference. Performance can be measured through various metrics, such as classification accuracy or mean squared error, which are evaluated during inference. For example, the performance of the first trained AI/ML model can be determined by inputting real data samples into the first trained AI/ML model and comparing the outputs of the first trained AI/ML model (e.g., analytics predicted the first trained AI/ML model) to ground truth analytics values. Similarly, the performance of the second trained AI/ML model can be determined by inputting real data into the second trained AI/ML model and comparing the outputs of the second trained AI/ML model (e.g., analytics predicted the second trained AI/ML model) to ground truth analytics values. When evaluating the performance of two trained AI/ML models (e.g., the first and second trained AI/ML models to determine the usefulness of the synthetic data can be provided using binary values. In various embodiments, a value of 0 indicates that the synthetic data was not useful for training an AI/ML model (e.g., that the performance of the first AI/ML model is better than the performance, while a value of 1 indicates that the synthetic data was useful for training an AI/ML model. The same examples are applicable when the performance of a AI/ML model trained using only synthetic data and/or a combination of synthetic data and real data and the performance of an AI/ML trained using only real data

[00044] Accordingly, in various embodiments, in the absence of sufficient real data (e.g., enough real data to generate sufficiently accurate analytics or train an AI/ML model to generate sufficiently accurate predictions), synthetic data may be utilized in the generation of analytics and/or in AI/ML model training for improving the performance of the NWDAF. [00045] Although further detail will be provided below, briefly, described herein is a method for determining whether accuracy of analytics and/or an accuracy of a machine learning model (generally referred to as an AI/ML model) trained using real data (e.g., actual data) is improved by training the AI/ML model using synthetic data generated by a synthetic data generator (SDG). In various embodiments, both synthetic data and real data may be used by an NWDF to generate analytics and/or train an AI/ML model.

[00046] As described herein, for example, the network data analytics function (NWDAF) may include a capability to (e.g., the NWDAF may be configured to) determine an accuracy of analytics generated by the NWDAF and/or an accuracy of an AI/ML learning (ML) model trained by the NWDAF. The NWDAF may provide accuracy information indicative of the accuracy of the analytics generated by the NWDAF and/or an accuracy of the AI/ML learning (ML) model trained by the NWDAF to a NF service consumer of an analytics service and/or model training service provided by the NWDAF for use for by internal processes of the NF ser vice consumer in response receipt of a request for analytics received from the NF service consumer and/or a request to train an AI/ML model. In a 5G network, for example, the NWDAF, in various embodiments, may include several logical functions, including an analytics logical function (AnLF) configured for providing analytics services and generating analytics (e.g., statistics or predictions using, for example a trained AIML model configured to generate analytics) and a model training logical function (MTLF) configured for providing model training services to train AI/ML models. In various embodiments, the AnLF and MTLF may have accuracy of analytics checking and monitoring capabilities. For example, the MTLF may capabilities perform accuracy checking (e.g., determine an accuracy of) analytics generated by the AnLF, for example analytics (e.g., prediction generated by) a trained AI/ML model by determining a ratio of the number of correct predictions generated by the trained AI/ML model to the total number of predictions generated by the trained AI/ML model. The number of correct predictions generated by the trained AI/ML model is determined by comparing each respective prediction generated by the trained AI/ML model to its corresponding ground truth, (e.g., the corresponding true observed events in the network 100).

[00047] As mentioned above, the availability of data for generating analytics and/or for AI/ML model training (e.g., for training a AI/ML model) may determine a level of the accuracy of the generated analytics and/or a level of accuracy of a trained AI/ML model, and an amount of time (e.g., latency) the NWDAF requires to generate analytics and/or train an AI/ML model. In the absence of enough historical real data, synthetic data may be utilized by the NWDAF to generate analytics and/or used to train an AI/ML model for improving the accuracy of the analytics generated by the NWDAF (e.g., predictions generated by a trained AI/ML model) and/or improving the accuracy of the trained AI/ML model (e.g., the AI/ML model that results from AI/ML model training). Accordingly, ensuring the quality and usefulness of synthetic data may aid in improving the accuracy of the generated analytics and/or the accuracy of the trained AI/ML model.

[00048] Accordingly, methods and apparatuses for generation of high quality and useful synthetic data (e.g., synthetic data that improves the accuracy of analytics generated by a NWDAF and/or improves an accuracy of an AI/ML model trained using synthetic data) is described herein.

[00049] As described above, using synthetic data to address the limitations of real- world data being scarce may introduce additional considerations. For example, not all synthetic data has is good quality data (e.g., follow characteristics and features (statistical properties, distribution, relationship, etc.) of the real data, no outliers, not noisy, etc.) and not all synthetic data may be useful for.

[00050] A synthetic data generator may produce (e.g., generate) synthetic data that has poor quality. For example, a synthetic data generator that includes a trained AI/ML model that generates synthetic data samples may have hyperparameters that are not well tuned, the AI/ML model may produce (e.g., generate) synthetic data that has poor quality because the AI/ML model may produce (e.g., generate) redundant and similar synthetic data (e.g., redundant and similar synthetic data samples). Synthetic data with poor quality may impact the performance of the network entity that is using the synthetic data (e.g., may impact the analytics generated by the NWDAF or may impact the training of an AI/ML model that is less accurate than an AI/ML model trained). There may also be instances where, even if the synthetic data has good quality (e.g., satisfies the characteristics of the synthetic data that is required by the network entity (e.g., NWDAF) that requested the synthetic data, the synthetic data may turn out to be not useful, thereby impacting the performance of NWDAF by decreasing the performance of the NWDAF (e.g. ). This may happen, for instance, when the characteristics selected by NWDAF were not well-tuned. When using synthetic data, the NWDAF may monitor the impact of synthetic data on its performance (e.g., the impact on the analytics generated by the NWDAF when the NWDAF uses an AI/ML model that is trained using a combination of real and synthetic data to predict analytics using real data. Accordingly, described herein is an evaluation of the impact of synthetic data on the outcome of analytics generated by an NWDAF and/or AI/ML model training performed by a NDWAF, and how to use a reliable evaluation for generation of more useful synthetic data.

[00051] In accordance with the brief description, FIGS. 2A-2D are diagrams depict examples of systems 200A, 200B, 200C, and 200D for synthetic data generation, according to one illustrated aspect of the disclosure. In various embodiments, each respective system for synthetic data generation depicted in FIGS. 2A-2D includes a synthetic data controller (SDC) and a plurality of synthetic data generators (SDGs). FIGS. 2A-2D also depict block diagrams of various example components of the network 100 of FIG. 1. A 5G network and/or 6G network may be described as an example of the network 100, and it is intended that aspects of the following description shall be applicable to other types of network systems, as well. The network 100 may operate in accordance with the signals and connections shown in FIG. 1 such that the UE 102 is in communication with the network system 100 through a radio access network (RAN 106). Additionally, the network system may be divided into user plane components and functions and control plane components and functions. Unless indicated otherwise, the terms “component”, “function”, and “service” may be used interchangeably herein, and they may refer to and be implemented by instructions executed by one or more processors.

[00052] Example functions of the components are described below. The example functions are merely illustrative, and it shall be understood that additional operations and functions may be performed by the components described herein. Additionally, the connections between components may be virtual connections over service-based interfaces such that any component may communicate with any other component. In this manner, any component may act as a service “producer,” for any other component that is a service “consumer,” to provide services for network functions.

[00053] In various embodiments described below in FIGS. 2A-2D, a synthetic data controller (SDC) of the systems 200A, 200B, 200C, and 200D may one or more synthetic data generators (SDG). In various embodiments, the SDC may receive feedback from the NWDAF after the NWDAF uses synthetic data generated by one or more SDCs for generating analytics and/or AI/ML model training (e.g. training an AI/ML model). This feedback may involve the NWDAF generating information and providing the information to the SDG. The information generated by the NWDAF may include information indicative of usefulness of the synthetic data for generating analytics and/or , analytics identification (ID), an identifier of NWDAF and statistics information such as a degree of improvement or degradation of the generated analytics caused by utilization of the synthetic data in generating analytics, and/or a degree of improvement or degradation of the AI/ML model that is trained using the synthetic data. In various embodiments, the feedback received from the NWDAF (e.g., the information generated and sent by the NWDAF to the SDC) may be utilized by the SDC to determine whether to store or dispose of synthetic data generated by a SDGs, may be utilized as a retrieval key for future request for generation of synthetic data, and to improve the quality and usefulness of synthetic data generated by SDGs. In various embodiments, an SDG may comprise an AI/ML model that generates synthetic data. The SDC may tune the parameters of the SDG by tuning the parameters of the AI/ML model that generates synthetic data.

[00054] In various embodiments, the SDC provides coordination among multiple SDGs, and selects one or more SDGs of the multiple SDGs to generate synthetic data based on information included in a request for synthetic data received from the NWDAF.

[00055] In various embodiments, the SDC may translate a request for synthetic data received from a NWDAF to SDG-readable requests (e.g., to a request to generate synthetic data that is readable by an SDG). The SDC may send SDG-readable requests to SDGs selected by the SDC to generate synthetic data. Each respective SDG-readable request may include information for generating the synthetic data that is requested by the NWDAF (e.g., information indicative of which AI/ML model to use to generate synthetic data (e.g., synthetic data samples) and a quantity of synthetic data to be generated by the SDGs). For example, a request for synthetic data received by the SDC from a NWDAF may include information related to the synthetic data that is requested by the NWDAF. The information related to the synthetic data that is requested by NWDAF may include a total size of the synthetic data, a range of values for the synthetic data, and/or statistical characteristics of the synthetic data. The SDC may determine which SDGs are to be used to generate the synthetic data that is requested by the NWDAF. The SDC may determine the number of synthetic data (e.g., the number of synthetic data samples) that are to be generated by each of the SDGs. In various embodiments, the SDC may determine which SDGs (e.g., which Al-based synthetic data generators, which rule-based synthetic data generators, and which simulation-based synthetic data generators) are to be used to generate the synthetic data that is requested by the NWDAF.

[00056] In various embodiments, the SDC may ensure that synthetic data (e.g., synthetic data samples) of suitable quality is generated by an SDG or the SDGs by ensuring that the information related to the synthetic data that is requested provided by NWDAF (e.g., included in the request for synthetic data sent by the NWDAF) is satisfied. For example, the SDG discards any synthetic data (e.g., any synthetic data sample) that is generated which does not satisfy the statistical characteristics of the synthetic data (e.g. discards any synthetic data samples that are too correlated, such as synthetic data that are too similar to each other). [00057] In various embodiments, the SDC may operate as a validation entity that assists the SDGs in tuning hyperparameters of AI/ML models of the SDGs that generate synthetic data by ensuring that the hyperparameters of the AI/ML models of the SDGs that generate synthetic data are well tuned and by monitoring the AI/ML models of the SDGs that generate synthetic data for issues, such as model collapse where the AI/ML model of SDGs generate synthetic data samples that are very similar or even identical to each other. By monitoring the SDGs, the SDC may determine to discard synthetic data (e.g., synthetic data samples) that are generated by different SDCs (e.g., generated by AI/ML models of different SDGs) that are identical to each other.

[00058] In various embodiments, the SDC may reject requests and/or provide alternatives when synthetic data with specified set of parameters cannot be generated by the SDGs (e.g., when SDC determines that the SDGs can only generate 100 high-quality data samples instead of 500 high-quality data samples), or when the request (or a portion thereof) does not conform to the expected formats that SDC is capable of processing, or if a request is received from an unauthorized consumer/client (e.g., NWDAF). There can be other factors that can cause an SDC to reject requests and/or provide alternatives to a consumer of the SDC. One skilled in the relevant art will recognize that aspects may be practiced without one or more of these specific details or with other methods.

[00059] In various embodiments, the SDC includes a data repository and the SDC store synthetic data generated by the SDGs. In these embodiments, the SDC may provide historical synthetic data to the consumer (e.g., an entity requesting synthetic data from the SDC, such as the NWDAF). In some embodiments, another network function, or an 0AM entity of the network 100 includes a data repository for storing synthetic data generated by SDGs. For example, an ADRF may include a data repository for storing synthetic data generated by SDGs.

[00060] In various embodiments, the SDC may determine whether to continue storing or delete historical synthetic data (e.g., synthetic data previously generated by SDGs and stored in a data repository). In some embodiments, a consumer of the SDC (i.e., NWDAF, AMF, SMF, etc.) may determine that the SDC should continue storing the historical synthetic data in the data repository or delete historical synthetic data stored in the data repository and may request or instruct the SDC to continue storing the historical synthetic data in the data repository or delete the historical synthetic data stored in the data repository. In some embodiments, the SDC may determine whether or based on operator configuration/policies (current practice/standard).

[00061] For example, FIG. 2A depicts an example of a core network 108A comprising an NWDAF 220, an SDC 226 and an SDG 227. In various embodiments, as shown in FIG. 2A, the NWDAF 220 may include the SDC 226 and the SDG 227. In the example shown in FIG. 2 A, the NWDAF 220 obtains real data from the UE 150, other NF s of the network and/or an 0 AM entity of the network 100) and synthetic data generated by the SDG 227 from the SDC and generate analytics based on the real data and based on a combination of the real data and the synthetic data.

[00062] FIG. 2B depicts another example of a core network 108B that includes the NWDAF 220, SDC 226 and multiple SDGs 227 (designated SDG #1, SDG #2, . . ., SDG #n). As shown in FIG. 2B, the SDC 226 and the SDGs 227 are external to the NWDAF 220 (e.g., the NWDAF 220 does not includes the SDC 226 and SDGs 227).

[00063] FIG. 2C depicts another example of a core network 108C that includes the NWDAF 220, a data collection coordination function (DCCF) 225, the SDC 226, and multiple SDGs 227 (designated SDG #1, SDG #2, . . ., SDG #n). As shown in FIG. 2C, the DCCF includes the SDC 226. In the example shown in FIG. 2C, the DCCF 225 collects real data (e.g., obtains real data) from data repositories (not shown that persons of skill in the art will understand and appreciate) and provides the collected real data to the NWDAF 220. The DCCF 225 may also collect synthetic data generated by the SDGs from the SDC 226 and provide the synthetic data to the NWDAF 220.

[00064] FIG. 2D depicts the NWDAF 220, a network exposure function (NEF) 215, the SDC 226 and multiple SDGs 227 (designated SDG #1, SDG #2, . . ., SDG #n). As shown in FIG. 2D, the core network 108D includes NWDAF 220 and the NEF 215, and the SDC 226 and SDGs 227 located outside the CN 108D as stand-alone entities and communicate with the NWDAF via the NEF 215. In various embodiments, the NEF 215 provides the NWDAF 220 with secure access to the SDC 226 so that the NWDAF can obtain synthetic data generated by the SDGs 227 from the SDC 226.

[00065] In accordance with the brief description, FIGS. 3A-3B are simplified block diagrams of example core networks that include a synthetic data repository (SDR), according to one illustrated aspect of the disclosure. In various embodiments the SDR stores synthetic data that is generated by the SDG and/or SDGs.

[00066] FIG. 3 A depicts a core network 108E comprising the NWDAF 220, an analytics data repository function (ADRF) 221, the SDC 226, multiple SDGs 227 (designated SDG #1, SDG #2, ..., SDG #n), and an SDR 230 for storing synthetic data (e.g., synthetic data samples) generated by the SDGs). As shown in FIG. 3 A, the ADRF 211 includes the SDR 230. It will be appreciated that the core network 108E may include other network functions, however, these network functions are not shown for ease of illustration. In various embodiments, the ADRF 211 allows consumers of the ADRF 211 to store synthetic data (e.g., synthetic data samples) in the SDR 230, retrieve synthetic data (e.g., synthetic data samples) from the SDR 230, and/or remove (e.g., delete) synthetic data (e.g., synthetic data samples) from the SDR 230. The ADRF 211 may also allow a consumer of the ARDF 211 (e.g., a NWDAF) to store analytics generated by the NWDAF, retrieve analytics stored in the ARDF 211, and remove (e.g., delete) analytics from the ADRF 211.

[00067] FIG. 3B depicts a core network 108F comprising the NWDAF 220, the SDC 226, multiple SDGs 227 (designated SDG #1, SDG #2, . . ., SDG #n), and an SDR 230. As shown in FIG. 3B, the SDC 226 includes the SDR 230. In other words, the CN 108E includes an entity that includes the functionality of both the SDC 226 and the SDR 230.

[00068] Accordingly, FIGS. 3A and 3B depict exemplary core networks 108E, 108F of a 5G network (e.g., 5G network 100) that include a SDR 230.

[00069] FIGS. 2A-2D and FIGS. 3A-3B are merely examples of different core networks of network 100 (e.g., core networks 108A-108F) that have different architectures), and variations to the core networks described herein are contemplated to be within the scope of the present disclosure. In embodiments, the network may include other entities and/or network functions not illustrated in FIGS. 2A-2D and FIGS. 3A-3B. In embodiments, the network 100 may not include every entity and/or network function illustrated in FIGS. 2A- 2D and FIGS. 3A-3B. In embodiments, the connections may be implemented with different connections than those illustrated in FIGS. 2A-2D and FIGS. 3A-3B. Such and other embodiments are contemplated to be within the scope of the present disclosure.

[00070] In accordance with the brief description, FIG. 4 is a diagram of an example embodiment of control plane signals and operations among an analytics service consumer (e.g., a NF that is a consumer of an analytics service provided by an NWDAF), the NWDAF, data sources, an SDC, and SDGs, according to one illustrated aspect of the disclosure. In various embodiments, the analytics service consumer, the NWDAF, the SDC, and the SDGs depicted in FIG. 4 may correspond to similar analytics service consumer, the NWDAF, the SDC, and the SDGs described above in FIGS. 1, 2A-2D, and 3A-3B. The following paragraphs will describe various signals and operations. It will be understood that a described signal may have associated operations and a described operation may have associated signals.

[00071] At operation 401, an analytics service consumer (e.g., a consumer of analytics services provided by NWDAF) transmits a request for analytics to the NWDAF. In some embodiments, the request for analytics sent by the analytics service consumer to the NWDAF may be a request for the analytics service consumer to subscribe to an AnalyticsSubscription service of the NWDAF to provide analytics to the analytics service consumer when analytics are generated by the NWDAF. In some embodiments, the request to subscribe to an analytics subscription service of the NWDAF comprises a Nnwdaf_AnalyticsSubscription_Subscribe message.

[00072] In some embodiments, the request for analytics sent by the analytics service consumer to the NWDAF may be a request for an Analyticsinfo service of the NWDAF to provide analytics generated by the NWDAF to the analytics service consumer. In some embodiments, the request for an Analyticsinfo service of the NWDAF to provide analytics generated by the NWDAF comprises an Nnwdaf_AnalyticsInfo_Request message.

[00073] At operation 402, real data that is required by the NWDAF (e.g., NWDAF 220) is gathered (e.g., obtained) by the NWDAF from real data sources (e.g., data sources that collected real data such as an access and mobility management function (AMF), application functions (AFs), other network function (NFs) of the network 100, or operations and management entity (0AM)) of the network 100. The real data may comprise data that is collected by Network Functions and provided by to the NWDAF (e.g., data related to events that observed by the network function of a 5G network (e.g., AMF, SMF, PCF, etc.)) and data collected by 0AM entity and provided to the NWDAF. Examples of data collected by a NF may comprise, for example, data related to a load of the NF (generally referred to as NF load of a NF), data related to a status of a NF (generally referred to as a NF status of a NF), data related to resource usage of the NF (generally referred to as NF resource usage). Examples of data collected by an 0AM entity may comprises data indicative of a throughput of data provided to a UE, data indicative of a delay in transporting data through the 5G network, data indicative of traffic volume in a cell of the 5G network, data indicative of traffic volume in a network slice of the 5G network, data indicative of a subscribed QoS, data indicative of a measured QoS, data indicative of locations of UE’s that are communicating with the 5G network, data indicative of a application identities, and/or data indicative of locations of applications.

[00074] At operation 403, the NWDAF gathers (e.g., obtains) synthetic data that is required by the NWDAF (e.g., NWDAF) by sending a request to SDC to obtain and provide synthetic data that is required by the NWDAF. At operation 404, the SDC sends a request (e.g.., SDG-readable request) to generate synthetic data to SDG#1 and the SDC performs quality checking to check the quality of the synthetic data generated by SDG#1 (e.g., checks or determines a quality of the synthetic data generated by SDG #1), At operation 405, the SDC sends a request (e.g.., SDG-readable request) to generate synthetic data to SDG#n and the SDC performs quality checking to check the quality of the synthetic data generated by SDG#n (e.g., checks or determines a quality of the synthetic data generated by SDG #n). In some embodiments, operations 403, 404 and 405 may occur in parallel. In some embodiments, synthetic data generation and quality checking of the generated synthetic data may be performed as described above in FIGS. 2A-2D. The SDC sends a request (e.g.., SDG-readable request) to each SDG (e.g., SDG#1 to SDG#n) selected by the SDC to generate synthetic data as described above.

[00075] At operation 406, the NWDAF trains a first AI/ML model that generates analytics using a training dataset that includes only the gathered (e.g., obtained) real data, and trains a second AI/ML model that generates analytics using a training dataset includes the gathered (e.g., obtained) real data and the gathered (e.g., obtained) synthetic data. The first and a second AI/ML models have the same number of layers and same number of parameters, however, the first and second AI/ML models have different values for their parameters because the first and second AI/ML models are trained using different training datasets (e.g., the first AI/ML model is trained using a training dataset that includes only real data and the second AI/ML model is trained using a training dataset that includes both real data and synthetic data.

[00076] At operation 407, the NWDAF determines a first accuracy of analytics predicted by the first trained AI/ML model (e.g., the first AI/ML model trained using a training dataset that includes only the gathered real data) and determines an accuracy of the second AI/ML model trained using a training dataset that includes both the gathered (e.g., obtained) real data and gathered (e.g., obtained synthetic data. The first accuracy of the first AI/ML model is determined by inputting new real data samples into the first trained AI/ML model and comparing the prediction generated (e.g., output) by the first AI/ML model for each respective new real data sample to a ground truth associated with the respective new real data sample. The second accuracy of the second AI/ML model is determined by inputting the new real data samples into the second trained AI/ML model and comparing the prediction generated (e.g., output) by the second AI/ML model for each respective new real data sample to the ground truth associated with the respective new real data sample.

[00077] At operation 407, the NWDAF monitors the first and second accuracies determined at operation 406 to determine whether there is an accuracy improvement from using synthetic data to train an AI/ML model that generates analytics. In various embodiments, the NWDAF compares the first accuracy of analytics predicted using the first trained AI/ML model to the second accuracy determined at operation 406 to determine which of the first and second AI/ML models is more accurate (e.g., to determine whether the first AI/ML model that is trained using only real data is more accurate than the second AI/ML model that is trained using a combination of real data and synthetic data). In some embodiments, the NWDAF provides information indicative of which of the first and second AI/ML models is more accurate for the synthetic data controller information to use to determine whether to store the synthetic data generated by one or more synthetic data generators, or discard the synthetic data generated by one or more synthetic data generators.

[00078] In various embodiments, the NWDAF determines the usefulness of using synthetic data to train an AI/ML model is determined by comparing the first accuracy of the first AI/ML model to the second accuracy of the second AI/ML model. In some embodiments, the NWDAF generates information indicative of which of the first machine learning model and the second machine learning model predicts more accurate analytics. In some embodiments, the NWDAF provides information indicative of usefulness of using synthetic data to train an AI/ML model.

[00079] In some embodiments, NWDAF, at operation 407, also tracks and monitors analytics generated using different combination of percentages between real and synthetic data. For example, the NWDAF may compare a third accuracy of analytics predicted by the second AI/ML model when the second AI/ML model is trained the use of the real data and a first percentage of the synthetic to a fourth accuracy of analytics predicted by the second AI/ML model when the second AI/ML model is trained using the real data and a second percentage of the synthetic data. In some embodiments, the NWDAF provides to the synthetic data controller information relating to the third accuracy and the fourth accuracy for the synthetic data controller information to use to determine whether to store the synthetic data generated by one or more synthetic data generators, or discard the synthetic data generated by one or more synthetic data generators.

[00080] At operation 408, the NWDAF transmits a response to the request for analytics to the to the analytics service consumer sent at operation 401. In the embodiments the request for analytics comprises a request for the analytics service consumer to subscribe to an AnalyticsSubscription service of the NWDAF, the NWDAF notifies the analytics service consumer of the analytics generated by the NWDAF. In other words, the NWDAF provides the analytics predicted by the first AI/ML model when the analytics predicted by the first AFML model are more accurate than the analytics predicted by the second AI/ML model, and the NWDAF provides the analytics predicted by the second AI/ML model when the analytics predicted by the second AI/ML model are more accurate than the analytics predicted by the first AI/ML model. In various embodiments, the NWDAF notifies by sending Nnwdaf^AnalyticsSubscriptionJNTotify message to the analytics service consumer . In the embodiments the request comprises a request for an Analyticsinfo service of the NWDAF to provide analytics generated by the NWDAF to the analytics service consumer, the NWDAF sends an Nnwdaf_AnalyticsInfo_Request_Reponse message that includes the analytics generated by the NWDAF (e.g., the analytics predicted by the first AI/ML model when the analytics predicted by the first AI/ML model are more accurate than the analytics predicted by the second AI/ML model, and the NWDAF provides the analytics predicted by the second AI/ML model when the analytics predicted by the second AI/ML model are more accurate than the analytics predicted by the first AI/ML model).

[00081] At operation 409, the NWDAF sends feedback to the SDC. In other words, the NWDAF sends information determined based on the usefulness of using synthetic data to train an AI/ML model at 407. In various embodiments, the NWDAF may send information indicative of the usefulness of the synthetic data, an identifier of analytics service of the NWDAF which used synthetic data is and/or information indicative of scenarios where the synthetic data is useful and/or not useful, and an identifier of the NWDAF. In various embodiments, the information determined by the NWDAF based on the accuracy monitoring and tracking performed at 40. The information may include the difference between first and second accuracies predicted by the NWDAF using real data and using real with synthetic data, results of using different real -synthetic data splits (e.g., percentages), and which split/percentage yields optimal results.

[00082] At operation 410, the SDC uses the information provided by the NWDAF) at step 409. In various embodiments, the SDC may use the information provided by the NWDAF to determine whether to store the synthetic data that is generated, dispose of the synthetic data that is generated, or relocate the synthetic data that is generated to other repositories closer to other entities (e.g., another NWDAF) where the synthetic data is useful. In various embodiments, the SDC may determine to use the information as a retrieval key for faster synthetic data provisioning for future request for synthetic data from the NWDAF.

[00083] In various embodiments, the SDC may utilize the information provided by the NWDAF to improve the synthetic data generation capabilities of the SDGs by tuning hyperparameters of AI/ML models of the SDGs that generate the synthetic data (e.g., the synthetic data samples). The SDC may also utilize the information provided by the NWDAF to improve the efficiency of synthetic data generation by determining the optimal SDG or set of SDGs that are to generate synthetic data.

[00084] The operations of FIG. 4 are merely illustrative, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the operations may include other operations not illustrated in FIG. 4. In embodiments, the operations may not include every operation illustrated in FIG. 4. In embodiments, the operations may be implemented in a different order than that illustrated in FIG. 4. Such and other embodiments are contemplated to be within the scope of the present disclosure. Persons of skill in the art will appreciate that, although various example components are described as perform various functions, other components may perform those functions described in FIG. 4.

[00085] The following describes operations of method performed by a network data analytics function. The operations include receiving a request for analytics from a network function service consumer. The network data analytics function obtains real data from a data source and obtains synthetic data from at least one synthetic data generator. The network data analytics function determines a first accuracy of a first machine learning model trained using a training dataset comprising the real data and determines a second accuracy for a second machine learning model trained using a training dataset comprising the real data and the synthetic data. The network data analytics function compares the first accuracy to the second accuracy to determine which of the first machine learning model and the second machine learning model is more accurate and transmits information indicating which of the first machine learning model and the second machine learning model is more accurate.

[00086] The following describes operations of a method performed by a synthetic data generator. The operations include receiving, from a network data analytics function, information relating to a comparison of a first accuracy of analytics predicted by a first machine learning model trained using a real data and a second accuracy of analytics predicted by a second machine learning model trained using real data and synthetic data, and based upon the second accuracy being greater than or equal to the first accuracy, storing the synthetic data and based upon the second accuracy less than to the first accuracy, discarding the synthetic data.

[00087] Referring now to FIG. 5, a block diagram of a network apparatus for a 5G NR cellular network is shown. In some embodiments, the network apparatus implements or includes one or more network functions of a core network, including the NWDAF. The network apparatus includes an electronic storage 510, a processor 520, a network interface 540, and a memory 550. The various components may be communicatively coupled with each other. The processor 520 may be and may include any type of processor, such as a single-core central processing unit (CPU), a multi-core CPU, a microprocessor, a digital signal processor (DSP), a System-on-Chip (SoC), a graphic processing unit (GPU), a tensor processing unit (TPU), and or any other type of processor. The memory 550 may be a volatile type of memory, e.g., RAM, or a non-volatile type of memory, e.g., NAND flash memory. The memory 550 includes processor-readable instructions that are executable by the processor 520 to cause the apparatus to perform various operations, including the operations mentioned herein, such as the operations of FIG. 4.

[00088] The electronic storage 510 may be and include any type of electronic storage used for storing data, such as hard disk drive, solid state drive, optical disc, and/or other non- transitory computer-readable mediums, among other types of electronic storage. The electronic storage 910 stores processor-readable instructions for causing or configured for causing the apparatus to perform its operations and also stores data associated with such operations, such as storing data relating to 5G network standards, among other data. The network interface 940 may implement wireless networking technologies such as 5G and/or other wireless networking technologies.

[00089] The components shown in FIG. 5 are merely examples, and persons skilled in the art will understand that an apparatus may include other components not illustrated and may include multiples of any of the illustrated components. Such and other embodiments are contemplated to be within the scope of the present disclosure.

[00090] Further embodiments of the present disclosure include the following examples.

[00091] Example 1.1. An apparatus comprising: a network analytics data function, NWDAF, comprising: means for receiving a request for analytics from a network function service consumer; means for receiving real data from a data source; means for receiving a synthetic data from at least one synthetic data generator; means for determining, by the network data analytics function, a first accuracy for analytics predicted by a first machine learning model trained using a training dataset comprising the real data; means for determining a second accuracy for analytics predicted by a second machine learning model trained using a training dataset comprising the real data and the synthetic data; means for comparing the first accuracy to the second accuracy and; means for determining which of the first machine learning model and the second machine learning model predicts more accurate analytics based on the comparison.

[00092] Example 1.2. The apparatus of example 1.1, wherein the means for comparing comprises means for determining that the first accuracy is less than the second accuracy, and sending a response to the request for analytics, the response comprising the analytics predicted by the second machine learning model.

Example 1.3. The apparatus of any of examples 1.1-1.2, wherein the means for comparing comprising means for the determining that the first accuracy is greater than the second accuracy, and sending a response to the request for analytics, the response comprising the analytics predicted by the first machine learning model.

Example 1.4. The apparatus of any of examples 1.1-1.3, wherein the NWDAF further comprises means for sending, to the synthetic data controller, information indicative of which of the first machine learning model and the second machine learning model predicts more accurate analytics.

[00093] Example 1.5. The apparatus of any of examples 1.1 - 1.4, wherein the means for determining of the second accuracy comprises means for determining a third accuracy of analytics predicted using a third machine learning model trained using a training dataset comprising the real data and a first percentage of the synthetic data, means for determining a fourth accuracy of analytics predicted using a fourth machine learning model trained using a training dataset comprising the real data and a second percentage of the synthetic data, means for comparing the third accuracy to the fourth accuracy, means for determining the second accuracy to be third accuracy when the third accuracy is greater than the fourth accuracy, and means for determining the second accuracy to be the fourth accuracy when the fourth accuracy is greater than the third accuracy.

[00094] Example 1.6. The apparatus of example 1.5, wherein the NWADF further comprises means for sending to the synthetic data controller information relating to the third accuracy and the fourth accuracy.

[00095] Example 1.7. The apparatus of any of examples 1.1-1.6, wherein the first data is from one or more of an access and mobility management function (AMF), application functions (AFs), or operations and management function (0AM).

[00096] Example 2.1. A synthetic data controller, comprising: means for receiving, by a synthetic data generator from a network data analytics function, information relating to a comparison of a first accuracy for a machine learning model and a second accuracy for the machine learning model; and means for, based upon the information, storing, by the first apparatus, the second data.

[00097] Example 2.2. The apparatus of example 2.1, wherein the first data is from one or more of an access and mobility management function (AMF), application functions (AFs), or operations and management function (0AM).

[00098] Example 2.3. The apparatus as in any one of examples 2.1-2.2, where the second data is from a synthetic data source.

[00099] Example 2.4. The apparatus of example 2.3, wherein, upon the second accuracy being greater than the first accuracy, storing, by the first apparatus, the synthetic data.

[000100] Example 2.5. The apparatus of example 2.3, wherein, upon the second accuracy being greater than the first accuracy, relocating, by the first apparatus, the synthetic data to a third apparatus.

[000101] Example 2.6. The apparatus of example 2.3, wherein, upon the first accuracy being greater than or equal to the second accuracy, disposing, by the first apparatus, the synthetic data.

[000102] Example, 2.7. The apparatus of example 2.1, wherein the determining of the second accuracy includes comparing a third accuracy including a use of a first percentage of the second data as the training data with a fourth accuracy including the use of a second percentage of the second data as the training data.

[000103] Example 2.8. The apparatus of example 2.7, wherein the first message includes information relating to the third accuracy and the fourth accuracy.

[000104] The embodiments and aspects disclosed herein are examples of the present disclosure and may be embodied in various forms. For instance, although certain embodiments herein are described as separate embodiments, each of the embodiments herein may be combined with one or more of the other embodiments herein. Specific structural and functional details disclosed herein are not to be interpreted as limiting, but as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure. Like reference numerals may refer to similar or identical elements throughout the description of the figures. [000105] For example, in various embodiments, the NWDAF provides analytics functions, as well as AI/ML model training/inferencing functions, as described above. However, any network function that provides analytics functions may be utilized.

[000106] The phrases “in an aspect,” “in aspects,” “in various aspects,” “in some aspects,” or “in other aspects” may each refer to one or more of the same or different aspects in accordance with this present disclosure. The phrase “a plurality of’ may refer to two or more. [000107] In various embodiments, the terms “first message” and “second message”, as well as any subsequent messages may refer to any messages that are transmitted or received in an order and are not necessarily limited to any particular message. In various terms the terms “first” and “second” may be used for a particular component, data, etc., but are used exemplary only, and do not necessarily imply an order.

[000108] The phrases “in an embodiment,” “in embodiments,” “in various embodiments,” “in some embodiments,” or “in other embodiments” may each refer to one or more of the same or different embodiments in accordance with the present disclosure. A phrase in the form “A or B” means “(A), (B), or (A and B).” A phrase in the form “at least one of A, B, or C” means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C) ”

[000109] Any of the herein described methods, programs, algorithms or codes may be converted to, or expressed in, a programming language or computer program. The terms “programming language” and “computer program,” as used herein, each include any language used to specify instructions to a computer, and include (but is not limited to) the following languages and their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, metalanguages which themselves specify programs, and all first, second, third, fourth, fifth, or further generation computer languages. Also included are database and other data schemas, and any other meta- languages. No distinction is made between languages which are interpreted, compiled, or use both compiled and interpreted approaches. No distinction is made between compiled and source versions of a program. Thus, reference to a program, where the programming language could exist in more than one state (such as source, compiled, object, or linked) is a reference to any and all such states. Reference to a program may encompass the actual instructions and/or the intent of those instructions.

[000110] While aspects of the present disclosure have been shown in the drawings, it is not intended that the present disclosure be limited thereto, as it is intended that the present disclosure be as broad in scope as the art will allow and that the specification be read likewise. Therefore, the above description should not be construed as limiting, but merely as exemplifications of particular aspects. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended hereto.

Claims

WHAT IS CLAIMED IS:

1. A method for a network data analytics function of a communication network, the method comprising: receiving a request for analytics from a network function service consumer; obtaining real data from one or more data sources of the communication network; determining, by the network data analytics function, a first accuracy of analytics predicted by a first trained machine learning model, wherein the first trained machine learning model is trained using a first training dataset comprising real data; obtaining synthetic data generated by one or more synthetic data generator from a synthetic data controller; determining, by the network data analytics function, a second accuracy of analytics predicted by a second trained machine learning model, wherein the second trained machine learning model is trained using a second training dataset comprising the real data and synthetic data; comparing, by the network data analytics function, the first accuracy to the second accuracy and; determining which of the first machine learning model and the second machine learning model predicts more accurate analytics based on the comparison.

2. The method as claimed in claim 1, wherein the comparing comprises determining that the first accuracy is less than the second accuracy, and sending a response to the request for analytics, the response comprising the analytics predicted by the second machine learning model.

3. The method as claimed in claim 1 or 2, wherein the comparing comprises determining that the first accuracy is greater than the second accuracy, and sending a response to the request for analytics, the response comprising the analytics predicted by the first machine learning model.

4. The method as claimed in any of claims 1 to 3, further comprising sending, to the synthetic data controller, information indicative of which of the first machine learning model and the second machine learning model predicts more accurate analytics.

5. The method as claimed in any of claims 1 to 4, wherein the determining of the second accuracy comprises determining a third accuracy of analytics predicted using a third machine learning model trained using a training dataset comprising the real data and a first percentage of the synthetic data, determining a fourth accuracy of analytics predicted using a fourth machine learning model trained using a training dataset comprising the real data and a second percentage of the synthetic data, comparing the third accuracy to the fourth accuracy, determining the second accuracy to be third accuracy when the third accuracy is greater than the fourth accuracy, and determining the second accuracy to be the fourth accuracy when the fourth accuracy is greater than the third accuracy.

6. The method as claimed in claim 5, further comprising sending to the synthetic data controller information relating to the third accuracy and the fourth accuracy.

7. The method as claimed in any of claims 1 to 6, wherein the one or more data sources comprise an access and mobility management function (AMF), application functions (AFs), or operations and management function (0 AM) of the communication network.

8. An apparatus, comprising: at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to perform method as claimed in any one of claims 1 to 7.

9. An apparatus comprising means for performing the method as claimed in any one of claims 1 to 7.

10. A non-transitory computer-readable medium storing instructions which, when executed by at least one processor of an apparatus, cause the apparatus at least to perform a method as in any one of claims 1 to 7.

11. A computer program comprising instructions which, when the computer program is executed by an apparatus, cause the apparatus at least to perform a method as in any one of claims 1 to 7.

12. A method comprising: receiving, by a synthetic data generator from a network data analytics function, information relating to a comparison of a first accuracy for analytics predicted using a first machine learning model trained using real data to a second accuracy for analytic predicted using a second trained machine learning model trained using a combination of real data and synthetic data; and based upon the information, determining by a synthetic data generator, whether to store or discard the synthetic data.

13. The method as claimed in claim 12, wherein the real data used to train the first machine learning model is from one or more of an access and mobility management function (AMF), application functions (AFs), or operations and management function (0AM).

14. The method as in claim 12 or 13, wherein the synthetic data used to train the second machine learning model is from a synthetic data source.

15. The method as claimed any of claims 12 to 14, wherein the determining comprises determining that the second accuracy is greater than the first accuracy and causing the synthetic data to be stored in a data repository.

16. The method as claimed any of claims 12 to 15, wherein the determining comprises determining that the first accuracy is greater than or equal to the second accuracy, discarding the synthetic data.

17. The method as claimed any of claims 12 to 16, wherein the determining of the second accuracy includes comparing a third accuracy including a use of a first percentage of the second data as the training data with a fourth accuracy including the use of a second percentage of the second data as the training data.

18. The method as claimed claim 17, wherein the information further comprises information relating to the third accuracy and the fourth accuracy.

19. A synthetic data generator, comprising: means for performing the method as claimed in any of claims 12 to 18.

20. A synthetic data generator, comprising: at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to perform a method as claimed in any of claims 12 to 18.

21. A non-transitory computer-readable medium storing instructions which, when executed by at least one processor of a synthetic data generator, cause the synthetic data generator at least to perform a method as in any one of claims 12 to 18.

22. A computer program comprising instructions which, when the computer program is executed by an apparatus, cause the apparatus at least to perform a method as in any one of claims 12 to 18.