EP4725168A1

EP4725168A1 - Synchronization based on quality of network digital twin

Info

Publication number: EP4725168A1
Application number: EP23736826.1A
Authority: EP
Inventors: Borislava GAJIC; Dario BEGA; Alberto Conte; Sina KHATIBI
Original assignee: Nokia Solutions and Networks Oy
Current assignee: Nokia Solutions and Networks Oy
Priority date: 2023-06-06
Filing date: 2023-06-06
Publication date: 2026-04-15
Also published as: CN121285986A; WO2024252172A1

Abstract

Method comprising: receiving, from a service consumer of a network digital twin, a request to provide an indication of a quality of the network digital twin; determining the quality of the network digital twin; providing the indication of the determined quality to the service consumer in response to the receiving the request, wherein the quality indicates how accurately the network digital twin represents a real network.

Description

SYNCHRONIZATION BASED ON QUALITY OF NETWORK DIGITAL TWIN

Technical Field

The present disclosure relates to a quality of a network digital twin.

Abbreviations

3GPP 3^rd Generation Partnership Project

5G/6G/7G 5^,h/6^,h/7^,h Generation

Al Artificial Intelligence

AIML Artificial Intelligence - Machine Learning

AnLF Analytics Logical Function

ES Energy Saving gNB Next Generation (5G) Node B

HO Handover

ID Identifier

KPI Key Performance Indicator

MDA Management Data Analytics

MDAS Management Data Analytics Service

ML Machine Learning

MLOps Machine Learning Operations

MM Mobility Management

MnS Management Service

NDT Network Digital Twin

NWDAF Network Data Analytics Function

OAM Operation and Maintenance

PRB Physical Resource Block

QoNDT Quality of Network Digital Twin

RRC Radio Resource Control

RSRP Reference Signal Received Power

RSRQ Reference Signal Received Quality

SINR Signal over Interference and Noise Ratio

SON Self-Organized Network

UE User Equipment Background

A digital Twin is a virtual instance of a physical system (twin) that is continuously updated with the latter's performance, maintenance, and health status data throughout the physical system's life cycle. Such updates are commonly referred as syncs (synchronizations) between the NDT and the physical/real system. Apart from getting the updated information from real system into a NDT, the opposite sync direction (from NDT towards real system) is also envisioned.

Digital twin network: a digital twin that is used in the context of networking. This is also called, digital twin for networks or network digital twin (NDT). ANDT may be used to answer “what/if” questions to estimate impact of certain measures (e.g. (re-)configurations) on the network, to develop recommendations and to collect feedback.

Summary

It is an object to improve the prior art.

According to a first aspect, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: receiving, from a service consumer of a network digital twin, a request to provide an indication of a quality of the network digital twin; determining the quality of the network digital twin; providing the indication of the determined quality to the service consumer in response to the receiving the request, wherein the quality indicates how accurately the network digital twin represents a real network.

The quality may be expressed by at least one of the following:

• an input drift indicating a difference between first data used to realize the network digital twin and corresponding data measured in the real network;

• an average of the input drift over a first time period;

• an output drift indicating a difference of second data generated by the network digital twin and corresponding data measured in the real network; or

• an average of the output drift over a second time period. The instructions, when executed by the one or more processors, may further cause the apparatus to perform receiving a specification of the at least one of the first data, the second data, the first time period, or the second time period; the determining by determining the at least one of the input drift or the average input drift for the specified first data or for the specified first time period, respectively, or of the output drift or the average output drift for the specified second data or for the specified second time period, respectively.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform receiving, from the service consumer, a first requirement for the quality; checking whether or not the determined quality fulfills the first requirement; synchronizing from the real network to the network digital twin in response to checking that the determined quality does not fulfill the first requirement; determining the quality after the synchronizing; the providing the indication of the determined quality such that the quality after the synchronizing is provided.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform at least one of the following in response to checking that the determined quality does not fulfill the first requirement:

- determining a time instance when the synchronizing from the real network to the network digital twin shall begin;

- determining a time instance when the synchronizing from the real network to the network digital twin shall end;

- determining a trigger event for initiating the synchronizing from the real network to the network digital twin;

- determining data to be updated in the network digital twin by the synchronizing from the real network to the network digital twin; or

- determining data of the real network to be used for the synchronizing from the real network to the network digital twin. The request to provide the indication of the quality of the network digital twin may comprise a request to perform a synchronization from the network digital twin to the real network; and the instructions, when executed by the one or more processors, may further cause the apparatus to perform receiving, from the service consumer, a second requirement for the quality; checking whether the determined quality fulfills the second requirement; synchronizing from the network digital twin to the real network according to the request to perform the synchronization in response to checking that the determined quality fulfills the second requirement; determining the quality after the synchronizing; the providing the indication of the determined quality such that the quality after the synchronizing is provided.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform at least one of the following in response to checking that the determined quality fulfills the second requirement:

- determining one or more actions to be performed by the synchronizing from the network digital twin to the real network;

- determining a time instance to perform the synchronizing from the network digital twin to the real network; or

- determining a trigger event and stopping the synchronizing from the network digital twin to the real network in response to detecting the trigger event.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform inhibiting the synchronizing from the network digital twin to the real network according to the request to perform the synchronization in response to checking that the determined quality does not fulfill the second requirement.

According to a second aspect, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the requesting, wherein the quality indicates how accurately the network digital twin represents a real network.

The quality may be expressed by at least one of the following:

• an average of the input drift over a first time period;

• an average of the output drift over a second time period.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform specifying the at least one of the first data, the second data, the first time period, or the second time period to the service producer.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform informing the service producer on a requirement for the quality.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform requesting the service producer to perform a synchronization from the network digital twin to the real network.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform checking whether the quality fulfills a criterion; determining an action on the real network based on a what-if analysis on the network digital twin; instructing a consumer of a management service to perform the determined action in response to checking that the quality fulfills the criterion.

The instructions, when executed by the one or more processors, may further cause the apparatus to perform inhibiting the determining the action on the real network in response to determining that the quality does not fulfill the criterion; or inhibiting the instructing the consumer of the management service to perform the determined action on the real network in response to determining that the quality does not fulfill the criterion.

The action may be related to at least one of mobility management in the real network or energy saving in the real network.

According to a third aspect, there is provided a method comprising: receiving, from a service consumer of a network digital twin, a request to provide an indication of a quality of the network digital twin; determining the quality of the network digital twin; providing the indication of the determined quality to the service consumer in response to the receiving the request, wherein the quality indicates how accurately the network digital twin represents a real network.

The quality may be expressed by at least one of the following:

• an average of the input drift over a first time period;

• an average of the output drift over a second time period.

The method may further comprise receiving a specification of the at least one of the first data, the second data, the first time period, or the second time period; wherein the determining may comprise determining the at least one of the input drift or the average input drift for the specified first data or for the specified first time period, respectively, or of the output drift or the average output drift for the specified second data or for the specified second time period, respectively.

The method may further comprise receiving, from the service consumer, a first requirement for the quality; checking whether or not the determined quality fulfills the first requirement; synchronizing from the real network to the network digital twin in response to checking that the determined quality does not fulfill the first requirement; determining the quality after the synchronizing; wherein the indication of the determined quality is provided such that the quality after the synchronizing is provided.

The method may further comprise at least one of the following in response to checking that the determined quality does not fulfill the first requirement:

- determining data of the real network to be used for the synchronizing from the real network to the network digital twin.

The request to provide the indication of the quality of the network digital twin may comprise a request to perform a synchronization from the network digital twin to the real network; and the method may further comprise receiving, from the service consumer, a second requirement for the quality; checking whether the determined quality fulfills the second requirement; synchronizing from the network digital twin to the real network according to the request to perform the synchronization in response to checking that the determined quality fulfills the second requirement; determining the quality after the synchronizing; wherein the indication of the determined quality is provided such that the quality after the synchronizing is provided.

The method may further comprise at least one of the following in response to checking that the determined quality fulfills the second requirement: - determining one or more actions to be performed by the synchronizing from the network digital twin to the real network;

The method may further comprise inhibiting the synchronizing from the network digital twin to the real network according to the request to perform the synchronization in response to checking that the determined quality does not fulfill the second requirement.

According to a fourth aspect, there is provided a method comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the method to perform: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the requesting, wherein the quality indicates how accurately the network digital twin represents a real network.

The quality may be expressed by at least one of the following:

• an average of the input drift over a first time period;

• an average of the output drift over a second time period.

The method may further comprise specifying the at least one of the first data, the second data, the first time period, or the second time period to the service producer.

The method may further comprise informing the service producer on a requirement for the quality. The method may further comprise requesting the service producer to perform a synchronization from the network digital twin to the real network.

The method may further comprise checking whether the quality fulfills a criterion; determining an action on the real network based on a what-if analysis on the network digital twin; instructing a consumer of a management service to perform the determined action in response to checking that the quality fulfills the criterion.

The method may further comprise inhibiting the determining the action on the real network in response to determining that the quality does not fulfill the criterion; or inhibiting the instructing the consumer of the management service to perform the determined action on the real network in response to determining that the quality does not fulfill the criterion.

Each of the methods of the third or fourth aspects may be a method of synchronization.

According to a fifth aspect, there is provided a computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to carry out the method according to any of the third or fourth aspects. The computer program product may be embodied as a non-transitory computer-readable medium or directly loadable into a computer.

According to some example embodiments, at least one of the following advantages may be achieved:

• Quality of the NDT may be specified (i.e., how accurately reflects NDT the real system);

• Synchronization schedules may be optimized such that synchronizations from real system to NDT may be performed when needed but unnecessary synchronizations from real system to NDT may be avoided. Synchronizations from NDT to real system may be performed only if there is high confidence that the actions and recommendations derived from NDT predict accurately the behaviour of the real system.

It is to be understood that any of the above modifications can be applied singly or in combination to the respective aspects to which they refer, unless they are explicitly stated as excluding alternatives.

Brief description of the drawings

Further details, features, objects, and advantages are apparent from the following detailed description of the preferred example embodiments which is to be taken in conjunction with the appended drawings, wherein:

Fig. 1 shows a message sequence chart according to some example embodiments;

Fig. 2 shows a message sequence chart according to some example embodiments;

Fig. 3 shows a message sequence chart according to some example embodiments;

Fig. 4 shows a message sequence chart according to some example embodiments;

Fig. 5 shows a message sequence chart according to some example embodiments;

Fig. 6 shows a message sequence chart according to some example embodiments;

Fig. 7 shows a message sequence chart according to some example embodiments;

Fig. 8 shows an apparatus according to an example embodiment;

Fig. 9 shows a method according to an example embodiment;

Fig. 10 shows an apparatus according to an example embodiment;

Fig. 11 shows a method according to an example embodiment; and

Fig. 12 shows an apparatus according to an example embodiment.

Detailed description of certain example embodiments

Herein below, certain example embodiments are described in detail with reference to the accompanying drawings, wherein the features of the example embodiments can be freely combined with each other unless otherwise described. However, it is to be expressly understood that the description of certain example embodiments is given by way of example only, and that it is by no way intended to be understood as limiting the disclosure to the disclosed details. Moreover, it is to be understood that the apparatus is configured to perform the corresponding method, although in some cases only the apparatus or only the method are described.

The Network Digital Twin should behave as close as possible to the real system that it represents, e.g. a single network function, set of network functions, or entire network, etc. However, it is usually not straightforward to create an identical replica of the real system’s object with all their dynamicity.

The network digital twin can be realized using different types of network data, e.g., historical data, real-time data, near-real time data or any combination of thereof. For example, the term historical data may refer to data collected in the past and that can be utilized to build the NDT model, and the term (near-)real time may refer to data that are being collected from the real mobile network, i.e. , almost live data. Thus, the difference between the NDT and the real-world object that it represents can largely differ depending on the data used to realize the NDT, the frequency with which the NDT is being synced with the real system as well as the complexity and computational cost of tools used to build the NDT. Correspondingly, the measure on “how good” the recommendations or decisions (e.g., on network re-configuration) derived within the NDT can be depends not only on the properties of the decision logic applied within NDT, but also on the quality/reliability of the data used to realize the NDT. Therefore, such quality/reliability of the data used to realize the NDT needs to be assessed, monitored and communicated to the NDT consumers.

Some example embodiments:

• quantify and monitor the difference between the NDT and the real system that it represents,

• provide the info on the difference between the NDT and the real system that it represents to the authorized consumers,

• perform the NDT management based on the difference between the NDT and the real system that it represents, o E.g. perform the updates/synchronizations between the NDT and the real system (in one or both directions between NDT and the real system).

Some example embodiments provide a metric for a Quality of the Network Digital Twin (NDT) i.e., QoNDT. Such metric reflects the authenticity/reliability/accuracy of NDT with respect to the real system, which it shall represent. It is the measure of diversion between the NDT and the ground truth in exact and quantifiable manner.

It is distinguished between the real system’s KPI(s) which is/are the KPIs/metrics/performance measurements collected for a given real system (e.g. network function, set of network functions or entire network), and a digital-twin-KPI(s) which represent(s) a KPIs/metrics/performance measurements generated by the NDT for the given real system. Using this distinction, the QoNDT is expressed by means of following metrics:

• Input drift: delta/drift in data used to realize the NDT compared to the measured data of the real system. The input drift can be given per: o time instance. o single input data instance (e.g. specific KPI/metric) used to realize the NDT, o a set of input data instances (e.g. set of specific most critical KPI/metric) o any combination thereof

• Average input drift: average delta/drift in data used to realize NDT compared to the measured data of real system over a given time period. The time period used to derive average input drift can be: o Consumer specified time window for which the average input drift is required o Producer specified time window o The period since last synchronization of the NDT based on the data from real system - this is the default value if no other value is specified by consumer of producer

• Output drift: delta/drift in data generated by the NDT compared to the measured data of real system i.e. , drift between the digital-twin-KPI and the real system KPI. The output drift can be given per: o time instance o single digital-twin-KPI data instance (e.g. specific KPI/metric) generated by the NDT o a set of digital-twin-KPI data instances (e.g. set of specific most critical KPI/metric generated by the NDT) o any combination thereof • Average output drift: average drift between the digital-twin-KPI and the real system KPI over a given time period. The time period used to derive average output drift can be: o Consumer specified time window for which the average output drift is required o Producer specified time window o The period since last synchronization of the NDT based on the data from physical system - this is the default value if no other value is specified by consumer of producer.

Some example embodiments allow the NDT service consumer (sometimes denoted “consumer” for the sake of simplicity) to monitor the QoNDT of an NDT. In particular, they allow the NDT service consumer to express their requirements on the monitoring in terms of metrics which shall be exposed and conditions under which the metrics shall be notified. Correspondingly, they allow the NDT service producer (sometimes denoted “producer” for the sake of simplicity) to report the current QoNDT to consumers based on the monitoring requirements of the respective consumer.

Some example embodiments allow a QoNDT driven synchronization between the NDT and the physical system it represents. This may comprise one or both of the following syncs: o real system towards NDT: determining the optimal schedule for performing the updates of the NDT with the data from real system (sync in direction from real system towards NDT) such that desired input and so output QoNDT are achieved

■ determine the need to perform the updates, e.g., due to QoNDT degradation

■ determine the time instance to perform the updates, e.g., based on the trend of QoNDT degradation

■ determine the KPI/metric/performance measurement which should be updated at NDT based on the data from physical system o NDT towards real system: determining the optimal schedule for performing the updates from the NDT towards the real system, i.e., determining the time instance for which the synchronization towards the real system is desired. For example, for a digital-twin-KPI, one may determine the time instance in which the output drift is minimal. This value of the digital-twin- KPI shall be used as input for decisions/recommendation, or which actions/recommendations on the real system may be performed based on NDT output

Hereinafter, several methods and procedures according to some example embodiments are explained at greater detail with reference to Figs. 1 to 7.

QoNDT monitoring and reporting

Fig. 1 illustrates QoNDT monitoring and reporting. The NDT service consumer may be a management entity, e.g. a SON function, or MDA service or an NWDAF-AnLF. The NDT service producer may be part of the network’s QAM system. The actions are as follows:

1 : NDT Service consumer issues the request to monitor a performance of specific NDT instance indicated by the NDTJD. In the monitoring request, the consumer may indicate which metrics should be provided, e.g., input/output drift and/or their average values. Optionally, the consumer may indicate the time window in which the average values shall be calculated. If nothing is specified, the producer may use default values.

2: NDT service producer provides the report on QoNDT metrics requested in Action 1. Optionally, the producer may provide the information on the time window used to calculate the average values. If nothing is specified, the default value is assumed.

Sync between the real system and NDT

Based on the QoNDT requirements specified by the consumer and the monitored actual QoNDT of the NDT instance, a sync between the real system and the NDT may be performed. There are two possible directions of sync between NDT and real system: a) Sync from real system to NDT - this sync typically happens if the QoNDT drops below a desired level indicated by the consumer. This implies that the input/output drift is larger than desired and the data used to realize the NDT, or the data produced by the NDT diverge compared to the real system more than desired. In such cases, new (up- to-date) data may be fetched from the real system and used to update the NDT. b) Sync from NDT to real system - this sync typically happens if the QoNDT meets the desired level indicated by the consumer. This implies that the input/output drift is lower than requested and the data used to realize the NDT , or the data produced by the NDT diverge compared to the real system less than allowed. In such cases the outcome of the NDT may reflect the behavior of the real system with high confidence, and thus it is desired to be used for actions and recommendations in the real system.

The desired levels for the sync from real system to NDT and the sync from NDT to the real system may be the same or different from each other. For example, in some example embodiments, in order to ensure high confidence in the actions and recommendations, the desired level of QoNDT for the sync from NDT to the real system may be higher than the desired level for the QoNDT triggering a sync from real system to NDT.

Sync from real system towards NDT

Fig. 2 illustrates actions for sync from real system towards NDT according to some example embodiments

1 : NDT service consumer may express towards the producer the requirements on desired QoNDT of the NDT instance. The request may include the information on desired input/output drift and/or their average values.

2: Based on the request received by the consumer and the actual monitored QoNDT, the producer determines the level of fulfilment of the QoNDT requirements. The sync from real system to NDT will be done if the actually monitored QoNDT is below the QoNDT required by the consumer. If the actually monitored QoNDT is above the QoNDT required by the consumer, sync from real system to NDT may be omitted.

The NDT service producer determines whether NDT updates (by performing the sync with real system) would be recommendable in view of the QoNDT requirement. The existing degradation of QoNDT below consumer specified requirements, or negative trend in monitored QoNDT may be triggers to determine that the NDT update is necessary.

3: In the case that sync from real system to NDT should be done the producer will decide on the actual sync details and the schedule. This may be done based on the actual monitored drift or the trend in the drift (e.g., drift increase over time) between NDT and real system. The sync details and schedule may include:

• Time instance in which the sync between NDT and real system shall begin • Time instance in which the sync between NDT and real system shall end

• Trigger event to initiate the sync between NDT and real system, e.g., reconfiguration of real network can be a trigger to sync the NDT

• Which input data shall be updated in NDT based on the real system KPIs/metrics/performance measurement. There could be the case in which the drift is caused by just one or a subset of input data to the NDT and not by all of them.

• What type of real system data shall be used for updates, e.g., o real time data, o near-real-time data (with specified maximal lag compared to real-time data) or o historical data (with specified maximal lag compared to real-time data)

• Etc.

4: Based on the schedule determined in Action 3, the sync from real system towards the NDT is performed. This includes taking the real system data as specified in the schedule, e.g., specific KPIs/metrics/performance measurement obtained by near-real time measurements of the real system and using them as input for realization of NDT. Typically, during the sync time (between the time instance in which the sync shall begin and the time instance in which the sync shall end) the output of NDT shall not be used, i.e. no sync from NDT towards real system shall be done.

5: After the performed sync the producer may report the new QoNDT metrics to the consumer

Sync from NDT towards real system

Fig. 3 illustrates the actions of the sync from NDT towards real system according to some example embodiments.

1 : NDT service consumer requests the NDT service producer to perform sync from NDT towards the real system. The request is under the condition that a requirement on QoNDT of the NDT instance is fulfilled. In particular, the request may include the information on desired input/output drift and/or their average values. 2: Based on the request received by the consumer and the actual monitored QoNDT, the producer determines the level of fulfilment of the QoNDT requirements. Depending on the fulfilment level, the sync from NDT to real system may be done - if the actual monitored QoNDT is above the QoNDT required by the consumer. Otherwise, if the QoNDT does not fulfill the requirements, the sync from NDT to real system may not be done.

In the example of Fig. 3, the NDT service producer determines that the quality of outputs from NDT is satisfying and that it can be used with high confidence for actions or recommendations towards the real system. The currently monitored QoNDT being above consumer specified requirements, or positive trend in monitored QoNDT may be triggers to determine that outputs of NDT may be used for actions and recommendation on the real system.

3: In the case that the QoNDT satisfies the consumer requirements, the outputs form NDT may be used for deriving actions and recommendations towards the real system (“sync from NDT to real system”). The producer will decide on the actual sync details and the schedule. This may be done based on the actually monitored drift or the trend in the drift (e.g., drift decrease over time) between NDT and real system. The sync details and schedule may include:

• Which actions on the real system may be performed based on the NDT output, e.g. in the case of reconfigurations which parameters may be reconfigured and how

• Which recommendation towards real system can be done based on the NDT output, e.g. which energy saving strategy may be applied, or which cells may be switched off for energy saving purpose

• Time instance in which the action or recommendation may be applied to real system

• Trigger event to stop the sync between NDT and real system, e.g., detected re-configuration of real network which was not recommended as outcome of NDT

• Etc.

4: Based on the schedule determined in Action 3, the sync from NDT towards the real system is performed. This may include reconfigurations of the real system based on the actions and recommendations derived from NDT.

5: After the sync from NDT to real system is performed, the producer may report the new QoNDT metrics to the consumer. With reference to Figs. 4 to 7, two specific use cases according to some example embodiments are explained at greater detail.

Example of use case specific NDT - mobility management

One of the commonly intended usages of Network digital twin is so called “what-if” types of analysis and network optimization, applying/training ML solutions in operational networks and towards network management automation. An example use case is mobility management. In this use case, "what-if" experiments may apply to HO parameters settings in network digital twin. So, NDT provides a digital replica of gNB configurations and environment model where the ML is applied in order to decide on the best HO decisions. The ML model runs in such NDT setup and provides inference results with some precision/confidence. The inference results may be for example the recommendation on the best target cell for performing HO to optimize mobility management. The ML model tests the proposed setting in the NDT and in case it results in better performance, the according configuration (e.g. HO settings to perform HO to recommended target cell) may be applied in the real network.

However, when applying the ML inference results to actual/real networks, the operator may take into account not only the ML KPIs (precision/confidence) but also the performance KPIs of the NDT (i.e. QoNDT) in which the “what-if” analysis has been conducted. In other words, the consumer of such ML inference needs to be aware on how “close” to real network the NDT is actually in order to be able to apply, with a certain confidence level, the results obtained in such NDT.

Figs. 4 and 5 illustrate how NDT is applied to mobility management use case. Fig. 4 illustrates a positive case, where the recommendations of the “what-if” analysis on the NDT are finally provided to the network management system to apply them on the network, while Fig. 5 illustrates a negative case, where the “what-if” analysis is not performed because of a QoNDT below the expectations.

The NDT service consumer, e.g., MDA (Management Data Analytics) assisted Mobility Management (MM) service (cf. 3GPP TS 28.104) takes into account different input parameters in order to provide the recommendations of optimal handover parameters to MDAS consumer, e.g. network management system. This MDA service may leverage the NDT as the source of needed input parameters, such as performance measurements (e.g., consumed physical and virtual resources) and UE measurements (MDT data) such as RSRP, RSRQ, SINR on serving and neighbouring cell, UE location information, etc. The actions shown in Fig. 4 are as follows:

1 : Since the MDA assisted MM relies on NDT data to produce its outputs, the MDA assisted MM service may request monitoring of QoNDT of the NDT (i.e. the MDA assisted MM plays the role of NDT Service Consumer). The MDA service may specify in its request which metrics should be reported, e.g., input and output drift of RSRP and SINR.

2: The NDT service producer measures the input and output drift on requested KPIs and provides the report. The input drift is measured evaluating the data distribution drift of the actual input data to the ML model compared with the data used during the most recent sync between the NDT and the real system. One possible approach to measure the output drift, and so, the accuracy of the NDT is to compare predictions/forecasts provided by the NDT with real data collected from the real networks: the NDT models/virtualizes the real network behaviour (or the specific aspects/entities/network elements that the NDT is capturing). This means that, starting from the actual network status, the NDT would be able to forecast the future status. If in the meanwhile no configuration updates are performed, the producer could measure the error between the forecasting and the real data once collected. This provides a measure of the NDT output drift. In the mobility management use case, the NDT could model the UEs interference and RSRP measured by the network cells in a certain network area. Thus, the NDT could forecast the expected future UEs interference and RSRP for all the cells and, if the network configuration does not change in the meanwhile (e.g., if no cells are switched on/off), one may evaluate the error with respect to the real data once collected. This provides a measure of the accuracy of the NDT and could be repeated periodically. Of course, during the measurement of the accuracy of the NDT, no configurations change should be applied, otherwise new forecast should be obtained from the NDT updating the new network status.

In the example of Fig. 4, QoNDT meets the requirements on QoNDT set by the NDT service consumer.

3: Based on the predictions/forecasts provided by the NDT, the MDA service may provide recommendations of optimal handover parameters. In addition, being able to monitor and quantize the accuracy of NDT, the MDA service is able to associate a confidence level to its recommendations. 4. The recommendations on handover parameters along with the assigned confidence values (based on the NDT accuracy) are provided to the MDA service consumer, e.g. network management system.

Fig. 5 illustrates the same use case as Fig. 4 but for a case that QoNDT is not sufficiently good. Actions 1 and 2 in Fig. 5 are the same as those in Fig. 4. However, since the report indicates that QoNDT is not sufficiently good, NDT service consumer does not perform a “what-if” analysis to derive the best HO cell (action 3). The confidence level of such an “what-if” analysis would be too low.

In the example of Fig. 4, action 3 (“Derive the best HO target cell”, the “what-if” analysis) is performed after receipt of the report from the NDT service provider indicating that the QoNDT is sufficiently good. In some example embodiments, instead, action 3 may be started before the report on QoNDT arrives at the NDT service consumer. However, in such example embodiments, the recommendation is provided to the network management system (action 4) only after receipt of the report indicating that QoNDT is sufficiently good. If the reported QoNDT is not sufficiently good, the recommendation is not provided to the network management system.

Example of use case specific NDT - energy savings

Another use case where NDT may be applied to optimize the decisions is energy saving. Figs. 6 and 7 illustrate the NDT applied to energy saving use case. Fig. 6 illustrates a positive case, where the recommendations of the “what-if” analysis on the NDT are finally provided to the network management system to apply them on the network, while Fig. 7 illustrates a negative case, where the “what-if” analysis is not performed because of a QoNDT below the expectations.

The NDT service consumer, e.g., MDA (Management Data Analytics) assisted Energy Saving (ES) service (cf. 3GPP TS 28.104) takes into account different input parameters in order to provide the recommendations of optimal energy saving decisions such as:

• Recommended cell to enter energySaving state.

• Recommended cells for taking over the traffic of switched off cell

• The time instances and load thresholds to enter and terminate the energy saving state This MDA service may leverage the NDT as the source of needed input parameters, such as performance measurements (e.g., Traffic load variation: PRB utilization rate; RRC connection number;) and UE measurements (MDT data) such as RSRP, UE location information, etc. The actions are as follows:

1 : Since the MDA assisted ES relies on NDT data to produce its outputs, the MDA assisted ES service requests monitoring of QoNDT of the NDT. The MDA service may specify in its request which metrics should be reported, e.g., input and output drift of traffic load (PRB utilization rate; RRC connection number) and UE location.

2: The NDT service producer measures the input and output drift on requested KPIs and provides the report. The input drift is measured evaluating the data distribution drift of the actual input data to the ML model compared with the data used during the most recent sync between the NDT and the real system. Similar to the mobility management use case, the NDT may forecast the expected future traffic load and UEs location and, if the network configuration does not change in the meanwhile (e.g., if no cells are switched on/off), it may evaluate the error with respect to. the real data once collected.

In the example of Fig. 6, the reported QoNDT is sufficiently good.

3: Based on the predictions/forecasts provided by the NDT, the MDA service provides the recommendations (e.g., cell to enter energysaving state, cells to take over the traffic of switched off cell, time instances, and/or load thresholds to enter and terminate the energy saving state). In addition, being able to monitor and quantize the accuracy of NDT, the MDA service is able to associate a confidence level to its recommendations.

4: The recommendations on energy savings along with the assigned confidence values (based on the NDT accuracy) are provided to the MDA service consumer, e.g. network management system.

Fig. 7 illustrates the same use case as Fig. 6 but for a case that QoNDT is not sufficiently good. Actions 1 and 2 in Fig. 7 are the same as those in Fig. 6. However, since the report indicates that QoNDT is not sufficiently good, NDT service consumer does not perform a “what-if” analysis to derive energy saving decisions (action 3). The confidence level of such a “what-if” analysis would be too low. In the example of Fig. 6, action 3 (“Derive the best energy savings decisions”, the “what-if” analysis) is performed after receipt of the report from the NDT service provider indicating that the QoNDT is sufficiently good. In some example embodiments, instead, action 3 may be started before the report on QoNDT arrives at the NDT service consumer. However, in such example embodiments, the decision is provided to the network management system (action 4) only after receipt of the report indicating that QoNDT is sufficiently good. If the reported QoNDT is not sufficiently good, the decision is not provided to the network management system.

Fig. 8 shows an apparatus according to an example embodiment. The apparatus may be a NDT producer or an element thereof. Fig. 9 shows a method according to an example embodiment. The apparatus according to Fig. 8 may perform the method of Fig. 9 but is not limited to this method. The method of Fig. 9 may be performed by the apparatus of Fig. 8 but is not limited to being performed by this apparatus.

The apparatus comprises means for receiving 110, means for determining 120, and means for providing 130. The means for receiving 110, means for determining 120, and means for providing 130 may be a receiving means, determining means, and providing means, respectively. The means for receiving 110, means for determining 120, and means for providing 130 may be a receiver, determiner, and provider, respectively. The means for receiving 110, means for determining 120, and means for providing 130 may be a receiving processor, determining processor, and providing processor, respectively.

The means for receiving 110 receives, from a service consumer of a network digital twin, a request to provide an indication of a quality of the network digital twin (S110). The quality indicates how accurately the network digital twin represents a real network. The quality may be a QoNDT.

The means for determining 120 determines the quality of the network digital twin (S120). The means for determining may determine the quality in response to receiving the request of S110, or independently from such a request (e.g. periodically).

The means for providing 130 provides the indication of the determined quality to the service consumer in response to the receiving the request of S110 (S130). Fig. 10 shows an apparatus according to an example embodiment. The apparatus may be a NDT consumer or an element thereof. Fig. 11 shows a method according to an example embodiment. The apparatus according to Fig. 10 may perform the method of Fig. 1 1 but is not limited to this method. The method of Fig. 11 may be performed by the apparatus of Fig. 10 but is not limited to being performed by this apparatus.

The apparatus comprises means for requesting 210 and means for receiving 220. The means for requesting 210 and means for receiving 220 may be a requesting means and receiving means, respectively. The means for requesting 210 and means for receiving 220 may be a requester and receiver, respectively. The means for requesting 210 and means for receiving 220 may be a requesting processor and receiving processor, respectively.

The means for requesting 210 request a service producer of a network digital twin to provide an indication of a quality of the network digital twin (S210). The quality indicates how accurately the network digital twin represents a real network. The quality may be a QoNDT.

The means for receiving 220 receives the indication of the quality in response to the requesting (S220).

Fig. 12 shows an apparatus according to an example embodiment. The apparatus comprises at least one processor 810, at least one memory 820 storing instructions that, when executed by the at least one processor 810, cause the apparatus at least to perform the method according to at least one of the following figures and related description: Fig. 9, or Fig. 1 1 .

Some example embodiments are explained with respect to 5G (NR). However, other example embodiments may be employed in other 3GPP generations, such as 4G, 6G, 7G, etc. Some example embodiments may be applied to non-3GPP communication networks (such as wired networks), or to networks which are different from communication networks, such as power grids.

A UE is an example of a terminal. Other examples are MTC devices. Each of the terminals may be implemented as a smartphone, a mobile phone, a laptop, a sensor device etc. One piece of information may be transmitted in one or plural messages from one entity to another entity. Each of these messages may comprise further (different) pieces of information.

Names of network elements, network functions, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and/or network functions and/or protocols and/or methods may be different, as long as they provide a corresponding functionality. The same applies correspondingly to the terminal.

If not otherwise stated or otherwise made clear from the context, the statement that two entities are different means that they perform different functions. It does not necessarily mean that they are based on different hardware. That is, each of the entities described in the present description may be based on a different hardware, or some or all of the entities may be based on the same hardware. It does not necessarily mean that they are based on different software. That is, each of the entities described in the present description may be based on different software, or some or all of the entities may be based on the same software. Each of the entities described in the present description may be deployed in the cloud.

According to the above description, it should thus be apparent that example embodiments provide, for example, a NDT service consumer or a component thereof, an apparatus embodying the same, a method for controlling and/or operating the same, and computer program(s) controlling and/or operating the same as well as mediums carrying such computer program(s) and forming computer program product(s). According to the above description, it should thus be apparent that example embodiments provide, for example, a NDT service consumer or a component thereof, an apparatus embodying the same, a method for controlling and/or operating the same, and computer program(s) controlling and/or operating the same as well as mediums carrying such computer program(s) and forming computer program product(s).

Implementations of any of the above described blocks, apparatuses, systems, techniques or methods include, as non-limiting examples, implementations as hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. Each of the entities described in the present description may be embodied in the cloud. It is to be understood that what is described above is what is presently considered the preferred example embodiments. However, it should be noted that the description of the preferred example embodiments is given by way of example only and that various modifications may be made without departing from the scope of the disclosure as defined by the appended claims.

The terms “first X” and “second X” include the options that “first X” is the same as “second X” and that “first X” is different from “second X”, unless otherwise specified. As used herein, “at least one of the following: <a list of two or more elements>” and “at least one of <a list of two or more elements>” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

Claims

Claims:

1 . Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: receiving, from a service consumer of a network digital twin, a request to provide an indication of a quality of the network digital twin; determining the quality of the network digital twin; providing the indication of the determined quality to the service consumer in response to the receiving the request, wherein the quality indicates how accurately the network digital twin represents a real network.

2. The apparatus according to claim 1 , wherein the quality is expressed by at least one of the following:

• an average of the input drift over a first time period;

• an average of the output drift over a second time period.

3. The apparatus according to claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform receiving a specification of the at least one of the first data, the second data, the first time period, or the second time period; the determining by determining the at least one of the input drift or the average input drift for the specified first data or for the specified first time period, respectively, or of the output drift or the average output drift for the specified second data or for the specified second time period, respectively.

4. The apparatus according to any of claims 1 to 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform receiving, from the service consumer, a first requirement for the quality; checking whether or not the determined quality fulfills the first requirement; synchronizing from the real network to the network digital twin in response to checking that the determined quality does not fulfill the first requirement; determining the quality after the synchronizing; the providing the indication of the determined quality such that the quality after the synchronizing is provided.

5. The apparatus according to claim 4, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform at least one of the following in response to checking that the determined quality does not fulfill the first requirement:

6. The apparatus according to any of claims 1 to 5, wherein the request to provide the indication of the quality of the network digital twin comprises a request to perform a synchronization from the network digital twin to the real network; and the instructions, when executed by the one or more processors, further cause the apparatus to perform receiving, from the service consumer, a second requirement for the quality; checking whether the determined quality fulfills the second requirement; synchronizing from the network digital twin to the real network according to the request to perform the synchronization in response to checking that the determined quality fulfills the second requirement; determining the quality after the synchronizing; the providing the indication of the determined quality such that the quality after the synchronizing is provided.

7. The apparatus according to claim 6, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform at least one of the following in response to checking that the determined quality fulfills the second requirement:

8. The apparatus according to claim 7, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform inhibiting the synchronizing from the network digital twin to the real network according to the request to perform the synchronization in response to checking that the determined quality does not fulfill the second requirement.

9. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the requesting, wherein the quality indicates how accurately the network digital twin represents a real network.

10. The apparatus according to claim 9, wherein the quality is expressed by at least one of the following:

• an average of the input drift over a first time period;

• an average of the output drift over a second time period.

11. The apparatus according to claim 10, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform specifying the at least one of the first data, the second data, the first time period, or the second time period to the service producer.

12. The apparatus according to any of claims 9 to 11 , wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform informing the service producer on a requirement for the quality.

13. The apparatus according to claim 12, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform requesting the service producer to perform a synchronization from the network digital twin to the real network.

14. The apparatus according to any of claims 9 to 13, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform checking whether the quality fulfills a criterion; determining an action on the real network based on a what-if analysis on the network digital twin; instructing a consumer of a management service to perform the determined action in response to checking that the quality fulfills the criterion.

15. The apparatus according to claim 14, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform at least one of inhibiting the determining the action on the real network in response to determining that the quality does not fulfill the criterion; or inhibiting the instructing the consumer of the management service to perform the determined action on the real network in response to determining that the quality does not fulfill the criterion.

16. The apparatus according to any of claims 14 and 15, wherein the action is related to at least one of mobility management in the real network or energy saving in the real network.

17. Method comprising: receiving, from a service consumer of a network digital twin, a request to provide an indication of a quality of the network digital twin; determining the quality of the network digital twin; providing the indication of the determined quality to the service consumer in response to the receiving the request, wherein the quality indicates how accurately the network digital twin represents a real network.

18. Method comprising: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the requesting, wherein the quality indicates how accurately the network digital twin represents a real network.

19. A computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to carry out the method according to any of claims 17 and 18.

20. The computer program product according to claim 19, embodied as a computer-readable medium or directly loadable into a computer.