EP4490892A1

EP4490892A1 - Trustworthiness of a chain of ai models

Info

Publication number: EP4490892A1
Application number: EP22713569.6A
Authority: EP
Inventors: Borislava GAJIC; Tejas SUBRAMANYA; Dario BEGA
Original assignee: Nokia Solutions and Networks Oy
Current assignee: Nokia Solutions and Networks Oy
Priority date: 2022-03-07
Filing date: 2022-03-07
Publication date: 2025-01-15
Also published as: WO2023169645A1

Abstract

Method comprising: monitoring if a metric function receives, from an intent/orchestration device, an overall requirement for a metric for a chain of plural artificial intelligence models; supervising if the metric function receives a configuration of the chain of the plural artificial intelligence models; if the metric function receives the overall requirement and the configuration, for each of the plural artificial intelligence models: deriving a respective individual requirement for the metric based on the overall requirement for the metric and the configuration of the chain; and checking if a metric value stored for the respective artificial intelligence model fulfills the respective individual requirement for the metric; if, for each of the plural artificial intelligence models, the respective stored metric value fulfills the respective individual requirement, informing the intent/orchestration device that the chain fulfills the overall requirement.

Description

Trustworthiness of a chain of Al models

Field of the invention

The present disclosure relates to chains of plural Al models. In particular, it relates to trustworthiness thereof.

Abbreviations

3GPP 3^rd Generation Partnership Project

5G/6G/7G 5^th/6^th/7^th Generation

ACK Acknowledge

AGV Automated Guided Vehicles

Al Artificial Intelligence

API Application Programming Interface

CAN Cognitive Autonomous Network

CNF Cognitive Network Function

E2E end-to-end eMBB enhanced Mobile Broadband

ID Identifier

KPI Key Performance Indicator

MANO Management and Orchestration

ML Machine Learning

MLFO ML Function Orchestrator

NACK Non-Acknowledge

O-RAN Open RAN

QoE Quality-of-Experience

QoS Quality-of-Service

QoT Quality-of-Trustworthiness

RAN Radio Access Network

RIC RAN Intelligent Controller

RT Realtime

SMO Service Management and Orchestration

TAI Trustworthy Artificial Intelligence

TAIF TAI Framework

TS Traffic Steering UE User Equipment

URLLC Ultra-reliable Low-latency Communication

Background

There are two main approaches in applying the AI/ML algorithms for solving a specific use case: applying a single model or a “chain” of modular AI/ML models [1], Which approach is the most appropriate depends on, e.g., complexity of the problem/use case, availability of AI/ML models and their suitability to solve the specific problem. For example, the customer intent in a Trustworthy AI/ML framework for Cognitive Autonomous Networks [PCT/EP2021/062396], cf. Fig. 1 , may not be realizable by a single AI/ML model, due to unavailability of the corresponding model. Instead, the customer intent may be realizable by combining different AI/ML models, i.e. , by building an AI/ML model chain.

Relying on a chain of models provides a more modular solution which also facilitates the reusability, easier replacement of models in the chain as well as separate evolvement, improvements and changes of the models in the chain. E.g., the better performing AI/ML model can be used (replacing the underperforming AI/ML model which was previously used in the chain) without the need for retraining or modifying other models in the chain, including the models that directly use the input of the newly inserted model. The drawback of such approach of a chain of modular models is that errors or performance degradation in chain components might be propagated through the chain, affecting the performance of other components and the overall performance of the chain.

A Trustworthy AI/ML framework for Cognitive Autonomous Networks was introduced in PCT/EP2021/062396, see Fig. 1. The Intent/Policy Manager translates the customer intent into Network QoS (e.g., Service Level Agreement), Al QoS (e.g., accuracy) & Al QoT (e.g., explainability) requirements and sends them to the Service Management & Orchestration (SMO), the Al Pipeline Orchestrator and the Al Trust Engine, respectively. Alternatively, the SMO may translate the Network QoS into Al QoS & Al QoT requirements and send them to the Al Pipeline Orchestrator and the Al Trust Engine, respectively. The Al Trust Engine can configure individual AI/ML models, monitor/collect Al performance and Al trust related metrics from the individual AI/ML models and, thus, verify if the required trustworthiness level of the individual AI/ML model is fulfilled. The term “trustworthiness” summarizes the parameters robustness, fairness, and explainability. Each of these parameters is an aspect of trustworthiness. Sometimes, the term “trustworthiness” may be related to only one or two of these aspects, sometimes the term “trustworthiness” is related to the three aspects.

1. Fairness: Fairness is the process of understanding bias introduced in the data, and ensuring the model provides equitable predictions across all demographic groups. It is important to apply fairness analysis throughout the entire AI/ML Pipeline, making sure to continuously re-evaluate the models from the perspective of fairness and inclusion. This is especially important when AI/ML is deployed in critical business processes that affect a wide range of end users. There are three broad approaches to detect bias in the AI/ML model: a. Pre-processing fairness - To detect bias in the AI/ML training data using algorithms such as Reweighing and Disparate impact remover. b. In-processing fairness - To detect bias in the AI/ML model generation using algorithms such as Prejudice Remover and Adversarial debiasing. c. Post-processing fairness - To detect bias in the AI/ML model decisions using algorithms such as Odds-equalizing and Reject option classification.

Quantification of Fairness - There are several metrics that measure individual and group fairness. For example, Statistical Parity Difference, Average Odds Difference, Disparate Impact and Theil Index.

2. Explainability: Explainability of an AI/ML model refers to unveiling of the black box model which just makes the prediction or gives the recommendation to the White box which actually gives the details of the underlying mechanism and pattern identified by the model for a particular dataset. There are multiple reasons why it is necessary to understand the underlying mechanism of an AI/ML model such as human readability, justifiability, interpretability and bias mitigation. There are three broad approaches to design an ML model to be explainable: a. Pre-modelling explainability - To understand or describe data used to develop AI/ML models. For example, using algorithms such as ProtoDash and Disentangled Inferred Prior Variational Autoencoder Explainer. b. Explainable modelling/lnterpretable modelling - To develop more explainable AI/ML models, e.g., ML models with joint prediction and explanation or surrogate explainable models. For example, using algorithms such as Generalized Linear Rule Models and Teaching Explainable Decisions (TED). c. Post-modelling explainability - To extract explanations from pre-developed AI/ML models. For example, using algorithms such as ProtoDash, Contrastive Explanations Method, Profweight, LIME and SHAP.

Furthermore, explanations can be local (i.e., explaining a single instance/prediction or global (i.e. , explaining the global AI/ML model structure/predictions, e.g., based on combining many local explanations of each prediction)).

Quantification of Explainability - While the concept of “explainability” suggests “quality of the explanation being evaluated by a human”, the research community has proposed quantitative metrics for explainability and to use explanations at the machine-level (as in [Hind]). There are several metrics that measure explainability such as Faithfulness and Monotonicity.

3. Robustness (adversarial): There are four adversarial threats that any AI/ML model developers/scientists need to consider defending and evaluating their AI/ML models and applications. a. Evasion: Evasion attacks involves carefully perturbing the input samples at test time to have them misclassified. For example, using techniques such as Shadow attack and Threshold attack. b. Poisoning: Poisoning is adversarial contamination of training data. Machine learning systems can be re-trained using data collected during operations. An attacker may poison this data by injecting malicious samples during operation that subsequently disrupt retraining. For example, using techniques such as Backdoor attack and Adversarial backdoor embedding. c. Extraction: Extraction attacks aim to duplicate a machine learning model through query access to a target model. For example, using techniques such as KnockoffNets and Functionally equivalent extraction. d. Inference: Inference attacks determine if a sample of data was used in the training dataset of an AI/ML model. For example, using techniques such as Membership inference black-box and attribute inference black-box. In addition to adversarial robustness, there are other aspects of AI/ML technical robustness such as dealing with missing data, erroneous data, confidence levels of data, fallback plans etc., which needs to be addressed.

There are a number of approaches to defend AI/ML models against such adversarial attacks at each stage of the AI/ML design: a. Preprocessor - For example, using techniques such as InverseGAN and DefenseGAN. b. Postprocessor- For example, using techniques such as Reverse sigmoid and Rounding. c. Trainer - For example, using techniques such as General adversarial training and Madry’s protocol. d. Transformer- For example, using techniques such as Defensive distillation and Neural cleanse. e. Detector - For example, using techniques such as Detection based on activations analysis and Detection based on spectral signatures.

Quantification of Robustness: There are several metrics that measure robustness of ML models such as Empirical Robustness and Loss Sensitivity.

In the remainder of this document, AI/ML models which are the building blocks of the chains may be referred to as modules or components (interchangeably).

An example implementation of a AI/ML chain for the use case for traffic steering (TS) may rely on chaining together two AI/ML modules. Namely the output of the QoS AI/ML module may be used as an input to the TS AI/ML module.

Another example is AGV QoS Prediction for Edge Cloud Offloading Optimization. Herein, the different modules (e.g., for route prediction, radio modelling, QoS prediction) are chained together in order to determine the most optimal decision on offloading the processing to the edge cloud.

The ITU-T defines the Machine Learning Function Orchestrator (MLFO) [2] which provides chaining functionality in terms of connecting ML nodes together to form an ML pipeline. The chaining performed by MLFO refers to connecting different nodes, with different functionalities in order to construct the ML Pipeline, e.g. a node which is the source of data (SRC node) with the node which is performing the collection (C) and pre-processing (PP) of data.

References

[1] 0-RAN.WG2.AIML-v01.03, Technical report from O-RAN Working Group 2 AI/ML workflow description and requirements

[2] Y3172 Architectural framework for machine learning in future networks including IMT-2020

Summary

It is an object of the present invention to improve the prior art.

According to a first aspect of the invention, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: monitoring if a metric function receives, from an intent/orchestration device, an overall requirement for a metric for a chain of plural artificial intelligence models; supervising if the metric function receives a configuration of the chain of the plural artificial intelligence models; if the metric function receives the overall requirement and the configuration, for each of the plural artificial intelligence models: deriving a respective individual requirement for the metric based on the overall requirement for the metric and the configuration of the chain; and checking if a metric value stored for the respective artificial intelligence model fulfills the respective individual requirement for the metric; if, for each of the plural artificial intelligence models, the respective stored metric value fulfills the respective individual requirement, informing the intent/orchestration device that the chain fulfills the overall requirement.

According to a second aspect of the invention, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: deciding if an overall requirement for a metric is to be fulfilled by a single artificial intelligence model or by a first chain of plural artificial intelligence models; informing a metric function on the overall requirement for the metric of the first chain of plural artificial intelligence models if the overall requirement is to be fulfilled by the first chain of plural artificial intelligence models; receiving, in response to the informing the metric function, an information whether or not the first chain fulfills the overall requirement.

According to a third aspect of the invention, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: monitoring if an information on a planned chain of plural artificial intelligence models including a first artificial intelligence model and a second artificial intelligence model subsequent to the first artificial intelligence model is received; checking if an output of the first artificial intelligence model matches an input of the second artificial intelligence model if the information on the planned chain is received; inhibiting configuring the planned chain if the output of the first artificial intelligence model does not match the input of the second artificial intelligence model.

According to a fourth aspect of the invention, there is provided a method comprising: monitoring if a metric function receives, from an intent/orchestration device, an overall requirement for a metric for a chain of plural artificial intelligence models; supervising if the metric function receives a configuration of the chain of the plural artificial intelligence models; if the metric function receives the overall requirement and the configuration, for each of the plural artificial intelligence models: deriving a respective individual requirement for the metric based on the overall requirement for the metric and the configuration of the chain; and checking if a metric value stored for the respective artificial intelligence model fulfills the respective individual requirement for the metric; if, for each of the plural artificial intelligence models, the respective stored metric value fulfills the respective individual requirement, informing the intent/orchestration device that the chain fulfills the overall requirement.

According to a fifth aspect of the invention, there is provided a method comprising: deciding if an overall requirement for a metric is to be fulfilled by a single artificial intelligence model or by a first chain of plural artificial intelligence models; informing a metric function on the overall requirement for the metric of the first chain of plural artificial intelligence models if the overall requirement is to be fulfilled by the first chain of plural artificial intelligence models; receiving, in response to the informing the metric function, an information whether or not the first chain fulfills the overall requirement.

According to a sixth aspect of the invention, there is provided a method comprising: monitoring if an information on a planned chain of plural artificial intelligence models including a first artificial intelligence model and a second artificial intelligence model subsequent to the first artificial intelligence model is received; checking if an output of the first artificial intelligence model matches an input of the second artificial intelligence model if the information on the planned chain is received; inhibiting configuring the planned chain if the output of the first artificial intelligence model does not match the input of the second artificial intelligence model.

Each of the methods of the fourth to sixth aspects may be a method of ensuring trustworthiness.

According to a seventh aspect of the invention, there is provided a computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to carry out the method according to any of the fourth to sixth aspects. The computer program product may be embodied as a computer-readable medium or directly loadable into a computer.

According to some example embodiments of the invention, at least one of the following advantages may be achieved:

• trustworthiness in a chain of Al models may be ensured;

• implementation of chains of Al models not fulfilling trustworthiness requirements may be avoided;

• a modular framework for chains of Al models is provided.

It is to be understood that any of the above modifications can be applied singly or in combination to the respective aspects to which they refer, unless they are explicitly stated as excluding alternatives. Brief description of the drawings

Further details, features, objects, and advantages are apparent from the following detailed description of the preferred embodiments of the present invention which is to be taken in conjunction with the appended drawings, wherein:

Fig. 1 shows a trustworthiness AI/ML framework for CAN according to the prior art;

Fig. 2 shows a trustworthiness AI/ML framework for CAN according to some example embodiments of the invention;

Fig. 3 (comprising Figs. 3a to 3c) shows a message sequence chart according to some example embodiments of the invention;

Fig. 4 shows an apparatus according to an example embodiment of the invention;

Fig. 5 shows a method according to an example embodiment of the invention;

Fig. 6 shows an apparatus according to an example embodiment of the invention.

Fig. 7 shows a method according to an example embodiment of the invention;

Fig. 8 shows an apparatus according to an example embodiment of the invention;

Fig. 9 shows a method according to an example embodiment of the invention; and Fig. 10 shows an apparatus according to an example embodiment of the invention.

Detailed description of certain embodiments

Herein below, certain example embodiments of the present invention are described in detail with reference to the accompanying drawings, wherein the features of the example embodiments can be freely combined with each other unless otherwise described. However, it is to be expressly understood that the description of certain example embodiments is given by way of example only, and that it is by no way intended to be understood as limiting the invention to the disclosed details.

Moreover, it is to be understood that the apparatus is configured to perform the corresponding method, although in some cases only the apparatus or only the method are described.

A specific problem/use case may be associated with a certain level of risk which implies that the related AI/ML solution needs to fulfil corresponding level of trustworthiness. However, the required trustworthiness level may not be achievable by chaining together AI/ML models. Thus, even if the performance requirements of the use case can be satisfied by a certain AI/ML model chain, there may be a need to verify if such AI/ML model chain is also compatible with the required overall trustworthiness level of the solution.

In the example of a 3GPP communication network, the AI/ML models (apart from having different performance and trustworthiness capabilities) may also be deployed in different domains (e.g. RAN, Core, Transport) and may be developed by the same or different entities/vendors. Currently, building an AI/ML model chain is case-by-case based, with tight coupling to specific use case implementation. Such approach is intrinsically inflexible, with low potential for automation and application to further use cases.

There is a need for a consistent domain-specific as well as cross-domain/E2E framework which allows the management of AI/ML model chains as well as the individual AI/ML model within a chain (i.e., modules of the chain) in extensible and automated way. Such framework shall comprise the means for building up and updating the chain, handling the chain requirements as well as module requirements, and matching them for realizing the specific intent for a use case. In particular, trustworthiness aspects of the AI/ML chain and fulfilment of trust requirements should be verified.

The above-mentioned implementations of AI/ML chains do not consider the trustworthiness aspects of the AI/ML chains or individual components of the chain, i.e. how the robustness, fairness, or explainability of individual AI/ML models may impact the robustness, fairness, or explainability of the overall chain.

The Al Trust Engine of [PCT/EP2021/062396] is not able to handle the trustworthiness of a chain of AI/ML modules. In particular, the following set of functionalities are lacking at the Al Trust Engine of [PCT/EP2021/062396]:

• the functionality to derive the module specific trustworthiness requirements and configuration based on trustworthiness requirements of the chain.

• the functionality to derive chain specific trustworthiness metrics based on the individual module specific metrics.

Similarly, the Al Pipeline Orchestrator is lacking corresponding functionalities for managing the performance requirements and fulfilment of AI/ML model chains. Each of trustworthiness and performance is a particular metric of a AI/ML model chain. Some example embodiments of the invention provide a logical function called e.g. “Al Chain Trust Function” (and associated APIs) which manages the trustworthiness aspects of the AI/ML model chain. Furthermore, an Al Chain Knowledge Database (a kind of a data repository) may be provided, which is responsible for storing the chain configurations, trustworthiness metrics, and further chain-related information. Optionally, in some example embodiments, a logical function called e.g. “Al Chain Orchestration Function” may be deployed for monitoring the performance metrics of the chain and report this and further information (e.g. configuration info) to the Al Chain Knowledge Database (cf. Fig. 2).

In contrast to the chain explained in [2], some example embodiments of the invention chain different models in order to solve more complex use cases. In some embodiments, further components of the pipeline, e.g., pre-processing (PP) or data collection (C) may be needed in addition between the chained models.

Hereinafter, without any loss of generality, sometimes models A, B, C are mentioned. It is assumed that model B receives at least part of its input from model A, and model C receives at least part of its input from model B. I.e., models A, B, and C form a chain “A-B-C”.

The Al Chain Trust Function may be a part of Service Management and Orchestration or Al T rust Engine and is responsible for at least one of the following operations:

• Disaggregation/breakdown of trustworthiness requirements: Derivation of individual, AI/ML module-specific requirements on trustworthiness (Al QoT requirements of individual modules in the chain). This procedure is based on the information on the required Al QoT for overall use case as derived by the Intent/Policy Manager and the information on the AI/ML models applied for specific use case, e.g. provided by Al Pipeline Orchestrator.

• Verifying if the required Al QoT for overall use case can be fulfilled, given the capabilities of the individual AI/ML models in the chain. This is performed by matching the individual AI/ML module-specific requirements on trustworthiness derived in a) with the queried/discovered trustworthiness capabilities of individual AI/ML models. The capabilities query has been detailed in the following patent applications: [explainability - PCT/EP2021/071153; fairness - PCT/EP2021/071156; adversarial robustness - PCT/EP2021/081004],

• Derive the configurations/instructions on which trustworthiness metrics of individual AI/ML models shall be measured or trustworthiness measures shall be applied. The measurement of the trustworthiness of one AI/ML model may or may not depend on measurements of the trustworthiness in other AI/ML models in the chain. There are at least two approaches: o Parallel approach: Measure all trustworthiness metrics simultaneously on all AI/ML models in the chain or apply trustworthiness methods to all AI/ML models simultaneously. E.g., for adversarial robustness verifying whether there is a possibility of data poisoning in each model of the chain and determining whether adversarial training needs to be applied to all the models in the chain in simultaneous way. o Event-based cascading approach: A measured metric or a certain quantity of the metrics from one AI/ML model (predefined condition) may serve as a trigger for starting measurements (or application of trustworthiness methods) on another AI/ML model in the chain. E.g. if models A and B are chained only if certain explanation is detected at model A start collecting explanations at model B, or only if the fairness of model A drops below some threshold, start measuring fairness at model B, etc.

• Monitoring of individual module trustworthiness metrics and collect the measurements

• Aggregation of module metrics: Derivation of aggregated/joint trustworthiness metrics of an overall chain, based on individual module trustworthiness metrics. E.g. o The sequence/relation of explainability metrics can be derived, following the way the individual models are chained: if models A and B are chained, the aggregated explanation may show the Model_B_output_explanation in relation to Model_A_output_explanation, where output of A has been used as input to B. E.g. traffic steering model (model B) inferred that the connection to the macro cell is the best, no handover to the pico cell shall be performed despite the increased load. This is because the mobility management module (model A) inferred the “short stay in a cell” if handover is performed. In this case, the chain explanation of QoE module (model C) would provide the explanation on the trade-off between the increased load which may still be acceptable and the risk of high signaling and delays, loss of connectivity etc. due to frequent handovers otherwise. o The mean fairness for specific input category across all chained models, e.g. mean weight applied for URLLC and eMBB slices related inputs. For example, this may be applicable if the second model in the chain is also receiving raw data as input and not only the output from the first model. o Robustness, e.g., for adversarial robustness verifying whether there is a possibility of data poisoning of the chain by verifying possibility of data poisoning of each model of the chain. For technical robustness quantifying the amount of missing data across all models in the chain, e.g., deriving the mean “missingness” ratio in the data.

• Performing further analytics and determining the model correlations and mutual dependencies, e.g. if/how the Al QoT of one model impacts the Al QoT of another model, etc. For example, if models A and B are chained, what is the probability that failures in model A (due to low technical or adversarial robustness) can result in failures of the model B.

• Extending the Al Chain Knowledge Database by providing the information on Al QoT of the individual AI/ML modules and the ability to fulfil the Al QoT requirements of the overall use case. Further derived information on dependencies among AI/ML modules can also be provided to the Al Chain Knowledge Database.

• Exposure of module-specific and/or aggregated and derived trustworthiness metrics to the interested consumer, based on the request. Note: Such exposure can be done directly from the Al Chain Knowledge Database, if available.

Optionally, the Al Chain Orchestration Function may be a part of Service Management and Orchestration or Al Pipeline Orchestrator and is responsible for at least one of the following operations: a) Determining if the customer’s intent and requested service is realized by a single AI/ML model or the chain of AI/ML modules is applied. The decision if the chain shall be applied may be performed at a different entity, e.g. SMO, Pipeline Orchestrator. In some example embodiments, the respective entity may have a choice among some pre-configured options, such as one or more single AI/ML models and one or more chains of AI/ML models. b) If the AI/ML chain shall be applied for realization of the service/intent, one or more of the following operations may be conducted: o Identifying the input/output matching, i.e., the output of which AI/ML module is used as input of which other AI/ML module. If there is a mismatch, the chain may not be implemented. o The identified information on AI/ML modules and input/output matching may be stored in the Al Chain Knowledge Database, as well. o Monitoring of individual module performance metrics o Aggregation of individual module performance metrics: Derivation of aggregated/joint performance metrics of an overall chain, based on individual module performance metrics o Correlation of performance metrics between individual modules in order to derive the level of dependencies between the modules. Performing further analytics and information derivation, e.g. which AI/ML modules are commonly used to build a chain, etc. o Exposure of module-specific and/or aggregated performance metrics to the interested consumer, based on the request. Note: Such exposure can be done directly from the Al Chain Knowledge Database.

Thus, the Al Chain Knowledge Database contains the learned mapping between the services and corresponding models/chains used to realize the service, along with models’/chains’ input/output parameters, QoS and QoT information and the suitability of the model to successfully realize the service.

Fig. 3 shows a message sequence chart according to some example embodiments of the invention. It illustrates actions for fulfilling the customer’s intent (translated into specific QoS and QoT requirements) in a chain of Al models. As an example use case, QoE fulfilment for service/slice may be considered. However, further mobile network related use cases, optimized management of network functions, i.e., optimized placement and offloading of network functions and processes, anomaly detection, diagnosis, and recovery, etc. are as well applicable. The Fig. 3 illustrates a case where the Al Chain Orchestration Function and the Al Chain Trust Function are part or the Al Pipeline Orchestrator and the Al Trust Engine respectively. However, similar procedures apply also in the case of collocation with the Service Management and Orchestration entity. Note that AI/ML models (modules) forming the chain may be provided/designed by different vendors or by a single vendor and can be deployed in a single domain or cross-domain. Procedure of continuous knowledge building and lookup:

In Fig. 3, the Al Chain Trust Engine is responsible for building and maintaining the Al Chain Knowledge Database by providing the information on Al QoT of the individual AI/ML modules as well as the aggregated Al QoT of the chain and achievability of required Al QoT by the Al chain.

Similarly, optionally the Al Chain Orchestration Function (as part of Al Pipeline Orchestrator or SMO) may extend the Al Chain Knowledge Database on Al QoS and input/output parameters of available AI/ML models used to realize the different services.

The information on Al QoS and Al QoT of individual models may be e.g., indicated by the model vendor, made available as part of the model description, and stored along with the model in the model repository, or discovered, learned and updated during the model operation by monitoring model trustworthiness and performance. The learning can rely on the information related to currently running AI/ML models or the models that have been running in the past (historical data).

Furthermore, the information on which AI/ML modules are chained together in order to solve specific use case/problem may be stored in the Al Chain Knowledge Database, as well. Such information may be used for identifying the dependencies and troubleshooting especially in more complex structure chains. The Al Chain Knowledge Database may further capture the information on the success of realizing a particular service/use case while applying a particular model or chain, with specific Al QoT and Al QoS.

Therefore, the Al Chain Knowledge Database contains the learned mapping between the services and corresponding models/chains used to realize the service, along with models’/chains’ input/output parameters, Al QoS and Al QoT information and the suitability of the model to successfully realize the service.

Table 1 illustrates information related to a single AI/ML model which the Al Chain Knowledge Database may contain according to some example embodiments of the invention. The Al Chain Knowledge Database may contain a corresponding entry for each of the AI/ML models.

Table 1 : Information per AI/ML model contained in Al Chain Knowledge Database

In the example embodiment of Fig. 3, the following actions are performed:

Action 1 : Customer intent is provided to the Intent/Policy Manager. Such intent may comprise the targets for a specific service, e.g. LIRLLC service, that should to be achieved, or information on the user categories (e.g., category description, UE association to different categories, etc.) for which the operator intends to achieve a particular QoE (e.g., expressed via a set of high- level goals or KPI targets that should to be achieved for particular UE groups).

Actions 2, 3: Intent/Policy Manager or the Service Manager and Orchestration translates the received intent into Al QoS (e.g., accuracy of models that shall be used) and Al QoT targets (e.g., with respect to model explainability) that should to be achieved. It informs the Al Pipeline Orchestrator and Al Trust Engine accordingly. The SMO also derives the network QoS targets (e.g., throughput or latency that shall be achieved) for the required service.

Action 4: The SMO or Al Pipeline Orchestrator (opt. with Al Chain Pipeline Orchestration Function) decides if a single model or a chain of AI/ML models shall be applied. The already available information on AI/ML model chains contained in Al Chain Knowledge Database may be used as one of the inputs in order to perform such decision. The decision may imply building a new chain or using a pre-defined/pre-configured chain suitable for the given use case. The decision may further require different levels of human operator involvements. The following information from the Al Chain Knowledge Database may be applicable in this action: e.g., which models/chains were used in past for realization of URLLC service/use case and target QoE, how successful was the realization of the service, and whether or not such models or pre-configured chains, with the requested Al QoS, are currently available. For example, the Al Chain Knowledge Database may contain information that the URLLC QoE use case may be achieved by a single model “mQoE” (small letter “m” here and hereinafter means “model”) as well as a chain of two models responsible for QoS monitoring and traffic steering “mQoS” and “mTS”. In the past, the chained approach has led to better overall network performance, and the pre-configured chain is already available. Therefore, the SMO or Al Pipeline Orchestrator may decide to proceed with the chained approach, e.g. apply the preconfigured chain. This is assumed in the explanation of the following actions.

Action 5: The Al Chain Pipeline Orchestration Function derives the individual, AI/ML modulespecific requirements on performance (Al QoS requirements of individual modules in the chain) based on the overall target Al QoS. There may be different approaches in derivation of performance requirements for individual modules (some of the implementation details may also be vendor specific), e.g. the accuracy of the overall model may be derived as a mean value of the accuracies of the individual modules, e.g., “mQoS” needs to have at least 95% accuracy whereas “mTS” needs to have at least 93% accuracy in order to achieve overall target of 94% accuracy for Al QoS of LIRLLC QoE use case.

The derived information on AI/ML modules as well as further known information e.g., input/output parameter matching from pre-configured chain may be stored in the Al Chain Knowledge Database, as well.

Action 6: The SMO or Al Pipeline Orchestrator (opt. with Al Chain Pipeline Orchestration Function) informs the Al Trust Engine about derived AI/ML chain components.

Action 7: The Al Chain Trust Function (as part of Service Management and Orchestration or Al Trust Engine) derives the individual, AI/ML module-specific requirements on trustworthiness (Al QoT requirements of individual modules in the chain) based on the chain QoT requirements. There may be different approaches in derivation of trustworthiness requirements for individual modules (some of the implementation details may also be vendor specific). E.g., if a “high” adversarial robustness is required for the overall chain due to potential data poisoning, this may imply performing adversarial trainings to all individual models of the chain such that the “high” adversarial robustness is achieved on each individual component, i.e. certain adversarial robustness requirements for overall chain may translate to the same adversarial robustness requirement for each component. Regarding a “high” required explainability for the overall chain this may translate e.g., into “high” explainability requirements in the last model of the chain, and at least “medium” explainability requirement in the remaining (precedent) models of the chain, e.g., in the above explained example, “mQoS” explainability shall be “medium”, and “mTS” explainability shall be “high”, to achieve a high explainability for overall QoE chain. Another example in order to achieve “high” explainability of overall chain, all components of the chain may need to fulfil “high” as well, etc. In order to derive such information, the Al Chain Trust Function may rely on the information stored in the Al Chain Knowledge Database, e.g., on model QoT and chain QoT from past realizations of the chain. Once the Al QoT of the individual AI/ML is derived, the Al Chain Trust Function may provide to the Al Chain Knowledge Database such information.

Action 8: Based on the trust capabilities of individual modules the Al Chain Trust Function verifies the achievability of required individual Al QoTs and provides the feedback (ACK/NACK) towards the SMO/AI Pipeline Orchestrator (opt. with Al Chain Orchestration Function) on the achievability of required Al QoT for the Al chain given by the Al Pipeline Orchestrator/SMO. E.g., the Al Chain Trust Function may send ACK if the “mQoS” can actually support medium explainability and “mTS” can actually support high explainability such that the desired high explainability of the overall chain may be achieved. This information can be also provided to the Al Chain Knowledge Database.

If the Al Chain Trust Function sends NACK (i.e., the chain cannot fulfil the trustworthiness requirements), the SMO/AI Pipeline Orchestrator may send an error message towards the Intent/Policy Manager and inhibit the implementation of the chain for the specific use case. On the other hand, the received NACK may serve as an indication to the SMO/AI Pipeline Orchestrator that (if available) a different chain shall be deployed which may be suitable for realization of a given use case. Such chain may comprise different AI/ML modules with different trustworthiness capabilities, which potentially may satisfy overall trustworthiness requirements. The NACK towards the Intent/policy Manager is sent only if none of the alternative chains available can address a given use case with required trustworthiness (or performance) levels.

Action 9: Derive the conditional configurations/instructions on which trustworthiness metrics of individual AI/ML models shall be measured and how the measurements of one individual AI/ML model are related with measurements of other AI/ML models in the chain. For example, it may select one of the following approaches:

• Parallel approach: Measure all trustworthiness metrics simultaneously on both “mQoS” and “mTS” models. E.g. explanations from both models shall be measured simultaneously, whereas the timestamp synchronization accuracy between the models may be less than X miliseconds.

• Event-based cascading approach: Measuring of a metric or a certain quantity of metrics from one AI/ML model may serve as a trigger for starting measurements on another AI/ML model in the chain. E.g. only if explanation “threshold for highest priority UEs will be reached in X seconds” of “mQoS” model is measured the data collection from “mTS” model shall start. This may decrease the amount of collected data, while ensuring that the most relevant data is captured.

Actions 10-11 : The Al Chain Trust Function monitors and reports the trustworthiness metrics of individual modules (“mQoS” and “mTS”) based on instruction derived in previous action.

Action 12: The Al Chain Trust Function may perform the aggregation of collected metrics and derive the trustworthiness metrics of the overall chain (QoE). It can perform further analytics, e.g. correlation of information in order to determine the dependencies of Al QoTs on different models and provide it to Al Chain Knowledge Database. In addition, it may expose such information to the interested consumer, based on the request/subscription. As another option, such exposure can be done directly from Al Chain Knowledge Database. Some examples of metrics aggregation are described below:

• The sequence and relation of explainability metrics can be derived, following the way the individual models are chained. E.g. “mQoS” inferred that the QoS for high priority UE1 will be above the threshold in next 8 hours due to expected low mobility (evening hours, UE usually staying at home, no commuting, etc.). Based on such explanation the “mTS” may still decide to steer the UE1 to pico cell in order to offload the macro cell for fast moving users. In this case the chain explanation of QoE module would provide the explanation on the offloading the macro cell in order to ensure capability for fast moving users during steady QoS level of UE1.

Furthermore, the Al Chain Trust Function may provide such aggregated metrics to the SMO/AI Pipeline Orchestrator. In the case of undesired resulting aggregated QoT, the SMO/AI Pipeline Orchestrator may provide (if available) a different chain that shall be deployed which may be suitable for realization of the given use case and resulting in the desired aggregated QoT. Such chain may comprise different AI/ML modules with different trustworthiness capabilities, which potentially may lead to the desired aggregated QoT. If such alternative is not available, the SMO/AI Pipeline Orchestrator may inhibit the AI/ML model chain.

Actions 13-15: The Al Chain Pipeline Orchestration Function monitors the performance of individual module performance and receives according performance metrics. Note: similarly to Al QoT conditional configurations of individual models, the parallel and event-based cascading approach can be applicable for configuring which Al model performance metrics shall be measured and how the measurements are related to measuring the performance of other models in the chain.

Action 16: The Al Chain Pipeline Orchestration Function may further aggregate the received metrics in order to derive chain specific metrics. Additionally, it may further process the received metrics, e.g., correlate the performance metrics among individual modules in order to derive the level of dependencies between the modules, e.g., what was actually achieved accuracy of “mQoS” and how it impacted the accuracy of “mTS”. This information can be provided to the Al Chain Knowledge Database. The Al Chain Pipeline Orchestration Function may optionally expose module-specific and/or aggregated performance metrics to the interested consumer, based on the request/subscription. As another option, such exposure can be done directly from Al Chain Knowledge Database.

Furthermore, the Al Chain Pipeline Orchestration Function may provide such aggregated metrics to the SMO/AI Pipeline Orchestrator. In the case of undesired resulting aggregated QoS, the SMO/AI Pipeline Orchestrator may provide (if available) a different chain that shall be deployed which may be suitable for realization of the given use case with the desired aggregated QoT. Such chain may comprise different AI/ML modules with different performance capabilities, which potentially may lead to desired aggregated QoS. If such alternative is not available, the SMO/AI Pipeline Orchestrator may inhibit the AI/ML model chain.

Finally, the SMO/AI Pipeline Orchestrator may provide the feedback towards the Intent Manger regarding the success/failure of the requested intent. The block of Actions 13-16 and the block of Actions 7-12 may be performed in an arbitrary sequence. They may be executed fully or partly in parallel.

Fig. 3 illustrates a process where the chain of models and the models are selected based on the performance requirements (actions 4 to 6), and the trustworthiness requirements are considered afterwards (actions 7 and 8). However, in some example embodiments, the chain of models and the models are selected based on the trustworthiness requirements, and the performance requirements are considered afterwards. The sequence may depend on the relevance of performance vs. trustworthiness. The higher relevant metric may be considered first. For example, based on the customer’s intent, intent/policy manager may decide on the sequence.

Fig. 4 shows an apparatus according to an example embodiment of the invention. The apparatus may be a Al chain metric function, such as a Al chain trust function or a Al chain performance function, or an element thereof. Fig. 5 shows a method according to an example embodiment of the invention. The apparatus according to Fig. 4 may perform the method of Fig. 5 but is not limited to this method. The method of Fig. 5 may be performed by the apparatus of Fig. 4 but is not limited to being performed by this apparatus.

The apparatus comprises means for monitoring 110, means for supervising 120, means for deriving 130, means for checking 140, and means for informing 150. The means for monitoring 110, means for supervising 120, means for deriving 130, means for checking 140, and means for informing 150 may be a monitoring means, supervising means, deriving means, checking means, and informing means, respectively. The means for monitoring 110, means for supervising 120, means for deriving 130, means for checking 140, and means for informing 150 may be a monitor, supervisor, deriver, checker, and informer, respectively. The means for monitoring 110, means for supervising 120, means for deriving 130, means for checking 140, and means for informing 150 may be a monitoring processor, supervising processor, deriving processor, checking processor, and informing processor, respectively.

The means for monitoring 110 monitors if a metric function (such as a chain trust function) receives an overall requirement for a metric for a chain of plural Al models (S110). It may receive the overall requirement from an intent manager or from an SMO/AI chain orchestrator or a combination of the intent manager and the SMO/AI chain orchestrator (“intent/orchestration device”). The metric may comprise at least one of a trustworthiness and a performance.

The means for supervising 120 supervises if the metric function receives a configuration of the chain of the plural Al models (S120).

S110 and S120 may be performed in an arbitrary sequence. They may be performed fully or partly in parallel.

If the metric function receives the overall requirement and the configuration (S110 = yes, and S120 = yes), the means for deriving 130 and the means for checking may perform for each of the plural Al models the following actions:

The means for deriving 130 derives, for each of the plural Al models, a respective individual requirement for the metric based on the overall requirement for the metric and the configuration of the chain (S130).

The means for checking 140 checks if a metric value stored for the respective Al model fulfills the respective individual requirement for the metric (S140).

If S130 and S140 are performed for each of the Al models and the respective stored metric value fulfills the respective individual requirement for each of the plural Al models (S140 = yes for each of the Al models), the means for informing 150 informs the intent/orchestration device that the chain fulfills the overall requirement (S150). If at least one of the Al models does not fulfill the respective individual requirement, the means for informing 150 may inform the intent manager that the chain does not fulfill the overall requirement. For example, one of the informing that the overall requirement is fulfilled and the informing that the overall requirement is not fulfilled may be implicit by not replying to the intent manager within a predefined time.

In some example embodiments, the block of S130 and S140 may be stopped as soon as one of the Al models does not fulfill the respective individual requirement for the metric. In this case, the overall requirement cannot be fulfilled, and the means for informing 150 may inform the intent manager accordingly before all the Al models are checked. Fig. 6 shows an apparatus according to an example embodiment of the invention. The apparatus may be an intent manager or orchestrator, or an element thereof. Fig. 7 shows a method according to an example embodiment of the invention. The apparatus according to Fig. 6 may perform the method of Fig. 7 but is not limited to this method. The method of Fig. 7 may be performed by the apparatus of Fig. 6 but is not limited to being performed by this apparatus.

The apparatus comprises means for deciding 205, means for informing 210, and means for receiving 220. The means for deciding 205, means for informing 210, and means for receiving 220 may be a deciding means, informing means, and receiving means, respectively. The means for deciding 205, means for informing 210, and means for receiving 220 may be a decider, informer, and receiver, respectively. The means for deciding 205, means for informing 210, and means for receiving 220 may be a deciding processor, informing processor, and receiving processor, respectively.

The means for deciding 205 decides if an overall requirement for a metric is to be fulfilled by a single Al model or by a first chain of plural Al models (S205).

If the overall requirement for the metric is to be fulfilled by the first chain of plural Al models, the means for informing 210 informs a metric function on the overall requirement for the metric of the first chain of plural Al models (S210). The means for receiving 220 receives, in response to informing the metric function, an information whether or not the first chain fulfills the overall requirement (S220). The metric may comprise at least one of a trustworthiness and a performance.

In some example embodiments, depending on whether or not the chain fulfills the overall requirement, the apparatus may or may not implement the chain.

Fig. 8 shows an apparatus according to an example embodiment of the invention. The apparatus may be a chain orchestrator or an element thereof. Fig. 9 shows a method according to an example embodiment of the invention. The apparatus according to Fig. 8 may perform the method of Fig. 9 but is not limited to this method. The method of Fig. 9 may be performed by the apparatus of Fig. 8 but is not limited to being performed by this apparatus. The apparatus comprises means for monitoring 310, means for checking 320, and means for inhibiting 330. The means for monitoring 310, means for checking 320, and means for inhibiting 330 may be a monitoring means, checking means, and inhibiting means, respectively. The means for monitoring 310, means for checking 320, and means for inhibiting 330 may be a monitor, checker, and inhibitor, respectively. The means for monitoring 310, means for checking 320, and means for inhibiting 330 may be a monitoring processor, checking processor, and inhibiting processor, respectively.

The means for monitoring 310 monitors if an information on a planned chain of plural Al models including a first Al model and a second Al model subsequent to the first Al model is received (S310). If the information on the planned chain is received (S310 = yes), the means for checking 320 checks if an output of the first Al model matches an input of the second Al model (S320). If the output of the first Al model does not match the input of the second Al model (S320 = no), the means for inhibiting 330 inhibits configuring the planned chain (S330). Otherwise, the apparatus may configure the planned chain.

Fig. 10 shows an apparatus according to an example embodiment of the invention. The apparatus comprises at least one processor 810, at least one memory 820 including computer program code, and the at least one processor 810, with the at least one memory 820 and the computer program code, being arranged to cause the apparatus to at least perform at least the method according to at least one of Figs. 5, 7, and 9 and related description.

Some example embodiments are explained with respect to a 5G network. However, the invention is not limited to 5G. It may be used in other mobile communication networks using beams, too, e.g. in previous of forthcoming generations of 3GPP networks such as 4G, 6G, or 7G, etc. It may be used in non-3GPP mobile communication networks using beams, too. It may be used in fixed networks. It may be used even in contexts outside communication networks such as factory automation or power grids.

Some example embodiments of the invention provide signalling that might be standardized as part of O-RAN.

The description and Fig. 2 provides some options where the logical units may be placed. However, the invention is not limited to these placements. In general, depending on the needs, the logical units may be placed anywhere. They may be implemented as stand-alone functions. In particular, logical units placed in the Intent/policy manager according to the above description may be placed in the SMO and vice versa, depending on implementation. Each of the intent/policy manager and SMO and their combination may be named “intent/orchestration device”.

One piece of information may be transmitted in one or plural messages from one entity to another entity. Each of these messages may comprise further (different) pieces of information.

Names of network elements, network functions, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and/or network functions and/or protocols and/or methods may be different, as long as they provide a corresponding functionality.

If not otherwise stated or otherwise made clear from the context, the statement that two entities are different means that they perform different functions. It does not necessarily mean that they are based on different hardware. That is, each of the entities described in the present description may be based on a different hardware, or some or all of the entities may be based on the same hardware. It does not necessarily mean that they are based on different software. That is, each of the entities described in the present description may be based on different software, or some or all of the entities may be based on the same software. Each of the entities described in the present description may be deployed in the cloud.

According to the above description, it should thus be apparent that example embodiments of the present invention provide, for example, a Al chain trust function or a component thereof, an apparatus embodying the same, a method for controlling and/or operating the same, and computer program(s) controlling and/or operating the same as well as mediums carrying such computer program(s) and forming computer program product(s). According to the above description, it should thus be apparent that example embodiments of the present invention provide, for example, an intent manager or a component thereof, an apparatus embodying the same, a method for controlling and/or operating the same, and computer program(s) controlling and/or operating the same as well as mediums carrying such computer program(s) and forming computer program product(s). According to the above description, it should thus be apparent that example embodiments of the present invention provide, for example, a Al chain orchestrator or a component thereof, an apparatus embodying the same, a method for controlling and/or operating the same, and computer program(s) controlling and/or operating the same as well as mediums carrying such computer program(s) and forming computer program product(s).

Implementations of any of the above described blocks, apparatuses, systems, techniques or methods include, as non-limiting examples, implementations as hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. Each of the entities described in the present description may be embodied in the cloud.

It is to be understood that what is described above is what is presently considered the preferred example embodiments of the present invention. However, it should be noted that the description of the preferred example embodiments is given by way of example only and that various modifications may be made without departing from the scope of the invention as defined by the appended claims.

The phrase “at least one of A and B” comprises the options only A, only B, and both A and B. The terms “first X” and “second X” include the options that “first X” is the same as “second X” and that “first X” is different from “second X”, unless otherwise specified.

Claims

Claims:

1. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: monitoring if a metric function receives, from an intent/orchestration device, an overall requirement for a metric for a chain of plural artificial intelligence models; supervising if the metric function receives a configuration of the chain of the plural artificial intelligence models; if the metric function receives the overall requirement and the configuration, for each of the plural artificial intelligence models: deriving a respective individual requirement for the metric based on the overall requirement for the metric and the configuration of the chain; and checking if a metric value stored for the respective artificial intelligence model fulfills the respective individual requirement for the metric; if, for each of the plural artificial intelligence models, the respective stored metric value fulfills the respective individual requirement, informing the intent/orchestration device that the chain fulfills the overall requirement.

2. The apparatus according to claim 1 , wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: if, for at least one of the plural artificial intelligence models, the respective stored metric value does not fulfill the respective individual requirement, informing the intent/orchestration device that the chain does not fulfill the overall requirement.

3. The apparatus according to any of claims 1 to 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: instructing a metric manager to measure an achieved individual value of the metric of at least one of the plural artificial intelligence models; receiving, for each of the at least one of the plural artificial intelligence models, an information on the respective achieved individual value of the metric; if the information on the achieved individual value of the metric is received for more than one of the plural artificial intelligence models: aggregating the respective achieved individual values of the metric to an achieved overall value of the metric; informing on at least one of the achieved individual values of the metric and the achieved overall value of the metric.

4. The apparatus according to claim 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform, if the metric manager is instructed to measure the achieved individual values of the metric of a subset of one or more of the plural artificial intelligence models and not instructed to measure the achieved individual values of the metric of the one or more artificial intelligence models not included in the subset: monitoring if the achieved individual values of the metric of one of the one or more artificial intelligence models of the subset fulfills a predefined condition; instructing the metric manager to measure the achieved individual value of the metric of at least one of the one or more artificial intelligence models not included in the subset if the achieved individual value of the metric of the one of the one or more artificial intelligence models of the subset fulfills the predefined condition.

5. The apparatus according to claim 4, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: for each of the one or more artificial intelligence models of the subset, querying a data repository for a relevance of the respective artificial intelligence model for the achieved overall value of the metric; selecting the one of the one or more artificial intelligence models of the subset according to its relevance for the achieved overall value of the metric.

6. The apparatus according to claim 3, wherein the instructing comprises instructing the metric manager to measure the achieved individual value of the metric of each of the plural artificial intelligence models; the receiving comprises receiving, for each of the plural artificial intelligence models, the information on the respective achieved individual value of the metric; and the informing comprises informing on the achieved overall value of the metric.

7. The apparatus according to any of claims 1 to 6, wherein the metric comprises at least one of a trustworthiness and a performance.

8. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: deciding if an overall requirement for a metric is to be fulfilled by a single artificial intelligence model or by a first chain of plural artificial intelligence models; informing a metric function on the overall requirement for the metric of the first chain of plural artificial intelligence models if the overall requirement is to be fulfilled by the first chain of plural artificial intelligence models; receiving, in response to the informing the metric function, an information whether or not the first chain fulfills the overall requirement.

9. The apparatus according to claim 8, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: inhibiting implementing the first chain if the first chain does not fulfill the overall requirement.

10. The apparatus according to any of claims 8 to 9, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: implementing the first chain if the first chain fulfills the overall requirement.

11 . The apparatus according to any of claims 8 to 10, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: selecting a second chain of plural artificial intelligence models to fulfill the overall requirement for the metric if the first chain does not fulfill the overall requirement; informing the metric function on the overall requirement for the metric of the second chain of plural artificial intelligence models if the overall requirement is to be fulfilled by the second chain of plural artificial intelligence models; receiving, in response to the informing the metric function, an information whether or not the second chain fulfills the overall requirement, wherein the second chain of plural artificial intelligence models is different from the first chain of plural artificial intelligence models.

12. The apparatus according to claim 11 , wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: for each of plural artificial intelligence models, querying a data repository for a relevance to achieve the overall requirement; wherein the instructions, when executed by the one or more processors, select the second chain of the plural artificial intelligence models according to the relevance of the artificial intelligence models to achieve the overall requirement.

13. The apparatus according to any of claims 8 to 12, wherein the metric comprises at least one of a trustworthiness and a performance.

14. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: monitoring if an information on a planned chain of plural artificial intelligence models including a first artificial intelligence model and a second artificial intelligence model subsequent to the first artificial intelligence model is received; checking if an output of the first artificial intelligence model matches an input of the second artificial intelligence model if the information on the planned chain is received; inhibiting configuring the planned chain if the output of the first artificial intelligence model does not match the input of the second artificial intelligence model.

15. Method comprising: monitoring if a metric function receives, from an intent/orchestration device, an overall requirement for a metric for a chain of plural artificial intelligence models; supervising if the metric function receives a configuration of the chain of the plural artificial intelligence models; if the metric function receives the overall requirement and the configuration, for each of the plural artificial intelligence models: deriving a respective individual requirement for the metric based on the overall requirement for the metric and the configuration of the chain; and checking if a metric value stored for the respective artificial intelligence model fulfills the respective individual requirement for the metric; if, for each of the plural artificial intelligence models, the respective stored metric value fulfills the respective individual requirement, informing the intent/orchestration device that the chain fulfills the overall requirement.

16. The method according to claim 15, further comprising: if, for at least one of the plural artificial intelligence models, the respective stored metric value does not fulfill the respective individual requirement, informing the intent/orchestration device that the chain does not fulfill the overall requirement.

17. The method according to any of claims 15 to 16, further comprising: instructing a metric manager to measure an achieved individual value of the metric of at least one of the plural artificial intelligence models; receiving, for each of the at least one of the plural artificial intelligence models, an information on the respective achieved individual value of the metric; if the information on the achieved individual value of the metric is received for more than one of the plural artificial intelligence models: aggregating the respective achieved individual values of the metric to an achieved overall value of the metric; informing on at least one of the achieved individual values of the metric and the achieved overall value of the metric.

18. The method according to claim 17, further comprising, if the metric manager is instructed to measure the achieved individual values of the metric of a subset of one or more of the plural artificial intelligence models and not instructed to measure the achieved individual values of the metric of the one or more artificial intelligence models not included in the subset: monitoring if the achieved individual values of the metric of one of the one or more artificial intelligence models of the subset fulfills a predefined condition; instructing the metric manager to measure the achieved individual value of the metric of at least one of the one or more artificial intelligence models not included in the subset if the achieved individual value of the metric of the one of the one or more artificial intelligence models of the subset fulfills the predefined condition.

19. The method according to claim 18, further comprising: for each of the one or more artificial intelligence models of the subset, querying a data repository for a relevance of the respective artificial intelligence model for the achieved overall value of the metric; selecting the one of the one or more artificial intelligence models of the subset according to its relevance for the achieved overall value of the metric.

20. The method according to claim 17, wherein the instructing comprises instructing the metric manager to measure the achieved individual value of the metric of each of the plural artificial intelligence models; the receiving comprises receiving, for each of the plural artificial intelligence models, the information on the respective achieved individual value of the metric; and the informing comprises informing on the achieved overall value of the metric.

21 . The method according to any of claims 15 to 20, wherein the metric comprises at least one of a trustworthiness and a performance.

22. Method comprising: deciding if an overall requirement for a metric is to be fulfilled by a single artificial intelligence model or by a first chain of plural artificial intelligence models; informing a metric function on the overall requirement for the metric of the first chain of plural artificial intelligence models if the overall requirement is to be fulfilled by the first chain of plural artificial intelligence models; receiving, in response to the informing the metric function, an information whether or not the first chain fulfills the overall requirement.

23. The method according to claim 22, further comprising: inhibiting implementing the first chain if the first chain does not fulfill the overall requirement.

24. The method according to any of claims 22 to 23, further comprising: implementing the first chain if the first chain fulfills the overall requirement.

25. The method according to any of claims 22 to 24, further comprising: selecting a second chain of plural artificial intelligence models to fulfill the overall requirement for the metric if the first chain does not fulfill the overall requirement; informing the metric function on the overall requirement for the metric of the second chain of plural artificial intelligence models if the overall requirement is to be fulfilled by the second chain of plural artificial intelligence models; receiving, in response to the informing the metric function, an information whether or not the second chain fulfills the overall requirement, wherein the second chain of plural artificial intelligence models is different from the first chain of plural artificial intelligence models.

26. The method according to claim 25, further comprising: for each of plural artificial intelligence models, querying a data repository for a relevance to achieve the overall requirement; wherein the instructions, when executed by the one or more processors, select the second chain of the plural artificial intelligence models according to the relevance of the artificial intelligence models to achieve the overall requirement.

27. The method according to any of claims 22 to 26, wherein the metric comprises at least one of a trustworthiness and a performance.

28. Method comprising: monitoring if an information on a planned chain of plural artificial intelligence models including a first artificial intelligence model and a second artificial intelligence model subsequent to the first artificial intelligence model is received; checking if an output of the first artificial intelligence model matches an input of the second artificial intelligence model if the information on the planned chain is received; inhibiting configuring the planned chain if the output of the first artificial intelligence model does not match the input of the second artificial intelligence model.

29. A computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to carry out the method according to any of claims 15 to 28.

30. The computer program product according to claim 29, embodied as a computer-readable medium or directly loadable into a computer.