WO2021099338A1 - Architecture pour réseau neuronal explicable - Google Patents

Architecture pour réseau neuronal explicable Download PDF

Info

Publication number
WO2021099338A1
WO2021099338A1 PCT/EP2020/082449 EP2020082449W WO2021099338A1 WO 2021099338 A1 WO2021099338 A1 WO 2021099338A1 EP 2020082449 W EP2020082449 W EP 2020082449W WO 2021099338 A1 WO2021099338 A1 WO 2021099338A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
explainable
network
explainable neural
layer
Prior art date
Application number
PCT/EP2020/082449
Other languages
English (en)
Inventor
Angelo Dalli
Mauro PIRRONE
Original Assignee
UMNAI Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by UMNAI Limited filed Critical UMNAI Limited
Priority to EP20811254.0A priority Critical patent/EP4062330A1/fr
Publication of WO2021099338A1 publication Critical patent/WO2021099338A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/045—Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions

Definitions

  • Neural networks have proven to be great at complex tasks, especially where a complicated relationship between a set of inputs and outputs needs to be found automatically.
  • a resurgence of useful AI algorithms that are outperforming humans has been fueled by the application of back- propagation-based methods such as Deep Learning.
  • neural networks lack transparency due to their inherent black-box architecture, leading to lack of trust, hidden biases and over-dependency on training data.
  • SENN proposes a method which enables the architecture of a neural network to have built-in interpretability in a bottom-up approach.
  • SENN enables point-wise local explanations which are achieved through a regularization scheme that ensures that the model not only looks like a linear model but also behaves like a linear model (locally). In such a scheme, data points which are close to each other should have similar coefficients.
  • a SENN consists of three components: a concept encoder that transforms the input into a small set of interpretable basis features, an input-dependent parametrizer that generates relevance scores, and an aggregation function that combines these scores to produce a prediction.
  • the robustness loss on the parametrizer encourages the full model to behave locally as a linear function on h(x) with parameters q(c), yielding immediate interpretation of both concepts and relevancies.
  • SENN explanations differ from typical explanation methods.
  • Concept based explanations may provide traditional input-based explanations.
  • the description of each concept is provided with an explanation. For instance, in a prediction, certain concepts or features may be more important than others.
  • AIM Additive Index Models
  • XNNs overcome the challenges in the present technologies by providing a precise and well- defined localization method that handles both local and global models.
  • the global modelling capabilities of XNNs make them superior to these technologies as XNNs can better handle the widest variety of AI and Machine Learning models possible.
  • XNNs also allow the contribution of specific data items to be identified precisely, while allowing the resulting activation paths within the network to also be understood fully.
  • XNNs are superior as they are a true and complete white- box approach that enable logically equivalent transformations to be applied, reducing the computational costs involved especially during runtime.
  • the computational cost optimization capabilities of XNNs are impossible to achieve with the current approaches.
  • XNN Explainable Neural Networks
  • XNN provide a global model that can zoom into an unlimited number of local partitions. Thus, they are simultaneously both global and local.
  • a precise and identifiable function for each of an XNN’s inherent components may exist.
  • An XNN may be computationally efficient in feed-forward execution mode, computing the result and the explanation in the same step, while also being efficient for future training passes using gradient descent methods and any other similar future training method that may be invented.
  • XNNs are a new type of Artificial Neural Network (ANN) that are inherently interpretable and explainable.
  • the main concept behind an XNN is that the inner network structure is fully interpretable without sacrificing model performance (i.e. accuracy and feed-forward execution speed). Interpretability is built within the architecture itself, yet it functions like a standard neural network. This eliminates the need to apply additional techniques or processing for interpreting the result of a neural network.
  • XNNs can utilize existing software infrastructures and hardware used for ANNs and may also remain fully compatible with back-propagation training techniques.
  • XNNs compute both the answer and its explanation in a single feed-forward step without any need for simulations, iterations, perturbation, etc.
  • XNNs are also designed to be easily and efficiently implementable both in software and hardware, leading to substantial speed and storage improvements.
  • the architecture behind an XNN combines multiple local models into one global model.
  • Local models analyze a small area within the entire search space. When a transaction is analyzed in a local manner, a linear model may be sufficient to explain the model.
  • global models provide a holistic view.
  • XNNs may merge the two - multiple partitions (or concepts) may represent the local zones and multiple linear models to explain each partition, while when combined they make up a global model. Additionally, XNNs go beyond linear data. They have been designed to cater to non-linear data by embedding transformations within the neural network itself, while still retaining explainability.
  • XNNs are the first ever known fully white-box ANNs, giving rise to a new category of neural networks that are understandable and interpretable.
  • an explainable neural network adapted to provide predictions and explanations
  • the explainable neural network comprising: an input layer configured for receiving inputs to the explainable neural network; a conditional network configured to receive said inputs from the input layer, wherein the conditional network is adapted to split said inputs into a plurality of partitions based on a set of rules; a prediction network configured to receive said inputs from the input layer, wherein the prediction network is adapted to fit at least one model associated with the plurality of partitions such that a combination of said at least one model forms a representative global model in relation to the plurality of partitions; a ranking layer configured to receive a combined output from the conditional network and prediction network; and an output layer configured to receive the combined output for providing the predictions and the explanations.
  • a second aspect is a computer-implemented method for training the explainable neural network according to any preceding claims to provide said predictions and explanations, the method comprising: receiving training data by an input layer of the explainable neural network, wherein the training data comprise at least one input and output pair; applying a gradient estimation algorithm to compute a gradient of a loss function with respect to weights of the explainable neural network for a pair input and output of the training data; and configuring the explainable neural network to update the weights with respect to the training data, wherein weights are updated using a deterministic or a heuristic optimization algorithm adapted to optimize the explainable neural network locally and globally.
  • a system for implementing an explainable neural network comprising: an input interface configured for receiving inputs to the explainable network; the explainable neural network configured according to the first aspect either directly or indirectly connected to the interface for receiving said inputs, wherein the explainable neural network is trained according to the second aspect; and a display interface configured for displaying predictions and explanations associated with said inputs.
  • a fourth aspect is a system for an artificial neural network that is interpretable and explainable, comprising: an input layer which receives an input and identifies one or more input features; a conditional network, comprising: a conditional layer configured to model the input features based on one or more partitions, wherein each of the one or more partitions comprises a rule; an aggregation layer configured to aggregate one or more rules into one or more of the partitions; and a switch output layer configured to selectively pool the aggregated partitions from the aggregation layer with the remaining partitions from the conditional layer; a prediction network, comprising: a feature generation and transformation network comprising one or more transformation neurons configured to apply one or more transformations to the input features; a fit layer configured to combine features which have been transformed by the feature generation and transformation network to identify one or more coefficients related to at least one of: one or more features and one or more partitions; a value output layer configured to output a value related to at least one of: one or more features, one or more partitions,
  • a fifth aspect is a computer implemented method for providing an explainable neural network, comprising executing on a processor the steps of: inputting a set of data into an input layer; partitioning the input based on one or more input features identified in the input, creating one or more partitions; aggregating one or more of the partitions; applying one or more transformation functions to the partitioned input features, providing a set of transformed features; combining the transformed features and identifying one or more coefficients corresponding to the transformed features; compiling one or more equations based on the coefficients and the transformed features; compiling one or more rules based on the equation and the partitions; applying the equations and rules to the set of input features to obtain an output value; and outputting the output value and outputting an explanation of the output value, wherein the explanation accounts for at least one of the coefficients.
  • the methods described herein may be performed by software in machine -readable form on a tangible storage medium e.g. in the form of a computer program comprising computer program code means adapted to perform all the steps of any of the methods described herein when the program is run on a computer and where the computer program may be embodied on a computer- readable medium.
  • tangible (or non-transitory) storage media include disks, thumb drives, memory cards etc. and do not include propagated signals.
  • the software can be suitable for execution on a parallel processor or a serial processor such that the method steps may be carried out in any suitable order, or simultaneously.
  • Figure 1 is a representation of an embodiment of a Self-Explaining Neural Network.
  • Figure 2 is a representation of an embodiment of an Explainable Neural Network using Additive Index Models.
  • Figure 3 is an exemplary embodiment of a method flowchart illustrating the steps involved in creating partitions.
  • Figure 4 is an exemplary embodiment of a flowchart illustrating the rule-based knowledge embedded in an XNN.
  • Figure 5 is an exemplary embodiment of a high-level XNN architecture.
  • Figure 6 is an exemplary diagram demonstrating a linear regression technique executable by a neural network.
  • Figure 7 is an exemplary diagram demonstrating a logistic regression technique executable by a neural network.
  • Figure 8 is an exemplary graph illustrating a linear fit and a polynomial fit on an exemplary data set.
  • Figure 9 is an exemplary flowchart illustrating an exemplary embodiment of the structure of a linear XNN model.
  • Figure 10 is an exemplary flowchart illustrating an exemplary embodiment of the structure of an XNN with multiple partitions and an output switch.
  • Figure 11 is an exemplary flowchart illustrating an exemplary embodiment of an XNN architecture.
  • Figure 12 is a Generalized High-Level XNN Architecture
  • Figure 13 Exemplary embodiment with partitions using the results of leaf nodes only
  • Figure 14 Exemplary embodiment with partitions using all hierarchical nodes
  • Figure 15 Exemplary embodiment with abstract concepts within partitions
  • Figure 16a Exemplary embodiment of the conditional network and prediction network combined together - using the result of the leaf nodes only
  • Figure 16b Exemplary embodiment of the conditional network and prediction network combined together - using the result of all nodes
  • Figure 17 Exemplary embodiment with concepts within partitions and taxonomy links
  • XNNs are a new type of ANN consisting of a combination of linear models (value network) combined with rules (conditional network). When multiple partitions are combined, they make up a globalized model. XNNs can be used both to predict values or for classification.
  • XNNs may provide valuable benefits over various existing techniques, such as the SENN (Self-Explaining Neural Network) architecture.
  • SENN Self-Explaining Neural Network
  • Figure 1 may illustrate a SENN architecture.
  • the SENN in Figure 1 includes an input 100, concept encoder 102, relevance parametrizer 104, aggregator 106, and finally, an explanation 108, provided as an output.
  • the input 100 may go through both the concept encoder 102 and the relevance parametrizer 104, which may respectively be used to define concepts 110 and relevancies 112.
  • the SENN may be burdened by a reconstruction loss and a classification loss .
  • the SENN may further be burdened with a robustness loss .
  • SENN applies a mathematical process which encourages the full model to behave locally as a linear function.
  • f(x) is a function which is trained end-to-end with back-propagation
  • h i represents the features
  • ⁇ i represents the feature importance.
  • h i typically represents the features directly from the input dataset
  • h i may represent a concept from a convolutional process.
  • the resulting output consists of an equation which gives the relevance scores for each concept.
  • Figure 2 may illustrate an Explainable Neural network using Additive Index Models (AIM).
  • AIM Additive Index Models
  • the architecture may consist of sub-networks which solve the AIM equation.
  • the equation g may represent a link function, where ⁇ is the intercept, are the projections indices of are the corresponding nonparametric ridge functions.
  • ⁇ is the intercept
  • Subnetworks may be used to learn the ridge functions, h j ( ⁇ ).
  • the internal structure of subnetworks may be less critical, provided that the subnetworks have sufficient structure to learn a broad class of univariate functions.
  • Subnetworks may consist of multiple fully-connected layers and use nonlinear activation functions.
  • the combination layer may be the final hidden layer of the explainable network and may consist of a single node.
  • the inputs of the node may be the univariate activations of all of the subnetworks.
  • the weights learned may provide a final weighting of the ridge functions.
  • a linear activation function may be used on this layer, so the output of the network as a whole may be a linear combination of the ridge functions.
  • interpretable and explainable may have different meanings.
  • Interpretability may be a characteristic that may need to be defined in terms of an interpreter.
  • the interpreter may be an agent that interprets the system output or artifacts using a combination of (i) its own knowledge and beliefs; (ii) goal-action plans; (iii) context; and (iv) the world environment.
  • An exemplary interpreter may be a knowledgeable human.
  • An alternative to a knowledgeable human interpreter may be a suitable automated system, such as an expert system in a narrow domain, which may be able to interpret outputs or artifacts for a limited range of applications.
  • a medical expert system or some logical equivalent such as an end-to-end machine learning system, may be able to output a valid interpretation of medical results in a specific set of medical application domains.
  • non-human Interpreters may be created in the future that can partially or fully replace the role of a human Interpreter, and/or expand the interpretation capabilities to a wider range of application domains.
  • model interpretability which measures how interpretable any form of automated or mechanistic model is, together with its sub- components, structure and behavior
  • output interpretability which measures how interpretable the output from any form of automated or mechanistic model is.
  • Model interpretability may be the interpretability of the underlying embodiment, implementation, and/or process producing the output, while output interpretability may be the interpretability of the output itself or whatever artifact is being examined.
  • a machine learning system or suitable alternative embodiment may include a number of model components.
  • Model components may be model interpretable if their internal behavior and functioning can be fully understood and correctly predicted, for a subset of possible inputs, by the interpreter.
  • the behavior and functioning of a model component can be implemented and represented in various ways, such as a state-transition chart, a process flowchart or process description, a Behavioral Model, or some other suitable method.
  • Model components may be output interpretable if their output can be understood and correctly interpreted, for a subset of possible inputs, by the interpreter.
  • An exemplary machine learning system or suitable alternative embodiment may be (i) globally interpretable if it is fully model interpretable (i.e. all of its components are model interpretable), or (ii) modular interpretable if it is partially model interpretable (i.e. only some of its components are model in terpre table). Furthermore, a machine learning system or suitable alternative embodiment, may be locally interpretable if all its output is output interpretable.
  • a grey-box which is a hybrid mix of a black-box with white-box characteristics, may have characteristics of a white-box when it comes to the output, but that of a black-box when it comes to its internal behavior or functioning.
  • a white-box may be a fully model interpretable and output interpretable system which can achieve both local and global explainability.
  • a fully white -box system may be completely explainable and fully interpretable in terms of both internal function and output.
  • a black-box may be output interpretable but not model interpretable, and may achieve limited local explainability, making it the least explainable with little to no explainability capabilities and minimal understanding in terms of internal function.
  • a deep learning neural network may be an output interpretable yet model un-interpretable system.
  • a grey-box may be a partially model interpretable and output interpretable system, and may be partially explainable in terms of internal function and interpretable in terms of output.
  • an exemplary grey-box may be between a white-box and a black-box on a scale of most explainable and interpretable (white-box) to least explainable and interpretable (black-box).
  • Grey- box systems may have a level of modular interpretability since some of their components may be model interpretable.
  • neural network architectures output interpretable results, but these results are not fully model interpretable. Hence, these neural network architectures are considered black-box, or grey-box if some inner components are considered model interpretable.
  • XNNs offer advantages over these neural network architectures, for example, by providing an output interpretable and fully model interpretable system (white -box). Unlike XNNs, none of these neural network architectures uses partitions to structure the model into well-defined and interpretable local models. Additionally, since they are only output interpretable, it may not be possible to encode the network into machine and human readable rules. Therefore, they may only provide local explanations, and may not be capable of providing global interpretability directly from the model structure. Other types of neural network architectures may have similar drawbacks.
  • XNNs may adopt a wide-network approach through the different partitions, thus enabling transformations to be simpler and easier to understand, rather than complex and impossibly opaque as with deep networks. It may further be contemplated that XNNs may also support inner components which make use of a deep network structure, thus in combination, superior over any existing neural network architectures.
  • XNNs While present technologies may use some form of a linear equation to model an explainable function using a neural network, XNNs differ in several aspects: First, XNNs use the notion of hierarchical features (or hierarchical concepts) through the notion of partitions and rules. In XNNs, the function f(x) can be seen as a local function, which models the feature importance within a specific partition. The combination of all the functions creates a global interpretable model. Further, XNNs may utilize an external process to identify suitable partitions during their initialization, yet also support further training using back-propagation and related technique. Current technology relies solely on using back-propagation to learn a function which may limit the level of explainability.
  • XNNs may be encoded into rules and may make use of an external process to initialize the structure of the XNN partitions, it is possible to embed human-knowledge within the neural network.
  • XNNs can mix human knowledge and machine -generated knowledge.
  • gradient descent methods such as back-propagation
  • knowledge in the form of rules could be refined to create a global model which can also function in a local manner.
  • the current technology is not capable of functioning as such, as a typical neural network is simply used to fit a single large function.
  • the weights and structure of an XNN can be encoded in a sequence of IF-THEN rules or disjunctive normal form (DNF) rules or Conjunctive normal form (CNF) rules.
  • DNF disjunctive normal form
  • CNF Conjunctive normal form
  • XNNs support rotational and scale invariance together with non-linear functions, and also summarization over multiple partitions in multiple dimensions.
  • the current technology uses a combination of linear functions and cannot represent non-linear data in the same manner that XNNs with a high-level of interpretability can.
  • XNNs provide a higher level of flexibility by enabling partitions to be customized according to the level of simplicity or complexity which is required. Presently, if the resulting model is interpretable, but still too complex to understand, there it is not possible to create new partitions in order to simplify the local interpretability, while still maintaining the global interpretability of XNNs.
  • An XNN being a true white -box system, does not suffer from any reconstruction losses or any classification losses.
  • a reconstruction step may not be needed by the XNN and a precise classification may be created as part of the explanation by the XNN in a lossless manner. All information needed to come up with the classification is preserved by the XNN, and additionally all the activation paths and methods used are also easily identifiable without any additional computational cost.
  • XNNs may be used to recognize hand-written digits.
  • the first step in the process of creating such an XNN would be to identify the partitions using a suitable clustering or hierarchical partitioning method, such as the induction method or a logical equivalent.
  • the clustering or hierarchical portioning method may utilize a variety of appropriate methods including, but not limited to, agglomerative clustering, divisive clustering, relocation partitioning, probabilistic clustering, k-mediod methods, k-means methods, fuzzy clustering, density based clustering, grid based methods, gradient descent based methods, evolutionary methods, region splitting, region growing, sub-space clustering, projection methods, co-clustering methods and lazy clustering methods.
  • the input to the partitioning method can be either the input features directly for low-dimensional data (i.e. tabular) or data which has been pre-processed (for example, from a convolutional network).
  • Features which have been transformed using a convolutional process may typically represent a higher-level of abstraction such as an edge, a stroke or a pattern.
  • image data could in theory be modelled as a flat list of pixels, a convolutional process is typically applied in order to reduce the dimensionality of the data while retaining the most important features. This makes neural networks more efficient to process and they typically converge to a better result.
  • convoluted features may include patterns of various numbers at different level of abstraction.
  • the input image may be convoluted 302 to higher-level abstracted features.
  • the feature data (or transformed feature data) may then be filtered 304 to determine which partition to activate. In this exemplary case the partition which distinguishes between 1 and 7 may be selected.
  • the prediction network finally ranks all the features of the convolutional network within the activated partition only 306.
  • the prediction network may determine that the convolutional filter which detects horizonal lines is the most important, which is why it has been used to classify the resulting output as number 7. If scale or rotational variances exist in the examples, the prediction network may automatically correct for this using non-linear methods.
  • an XNN variant such as a convolutional XNN (CNN-XNN) may be the most appropriate, due to the presence of convolution layers.
  • CNN-XNNs allow for explanation summaries to take into consideration the composition of the convolution filters and kernels used during each activation, thus allowing, for example, an object that has been identified using a diagonal edge kernel to be labelled as a “diagonal edge”. Therefore CNN-XNNs differ from XNNs which may provide a global model but can also behave like a local model through the partitioning mechanism.
  • the concepts or features in an XNN may be related to the activated partition(s) only.
  • XNNs may model a hierarchy of linear models together with support for non- linearities and rotational and scale invariances.
  • XNNs can also be extended and enriched with the addition of items such as taxonomical information, links to internal and external taxonomies, bias detectors, and also seamlessly integrated with knowledge bases and rule systems, all of which is impossible to do with existing approaches that are all black-box methods.
  • XNNs can also form part of a causal logic induction system that can extract higher level concepts and link them to a causal model, which is something that is completely impossible to do with a standard black-box neural network.
  • XNNs can link causal effects not just at the output layer but also within their internal layers, giving unique capabilities to the XNN architecture that allow AI systems using XNNs to achieve knowledge, understanding, and advanced reasoning capabilities outside the reach of deep learning based systems and machine learning techniques that do not fully support the entire range of first-order logic rules and assertions.
  • the XNN may embed rule -based knowledge in multiple formats.
  • a universal knowledge representation or compatible format may be implemented allowing for rules and related information to be easily encoded and transmitted over computer networks and also enable XNNs to work in offline modes and be easily implemented in hardware apart from software.
  • the representation format may be made up of a set of rules and assertions that have a localization trigger together with simultaneous processing for the answer and explanation production, which are then applied to domain specific applications, for example by transmitting and encoding the rules, knowledge and data for use in a medical diagnosis imaging scanner system so that it can produce a diagnosis, processed image and an explanation which can then be further used by other AI systems in an automated pipeline, while retaining human readability and interpretability.
  • the representation format may consist of a system of Disjunctive Normal Form (DNF) rules or other logical alternatives, like Conjunctive Normal Form (CNF) rules, first-order logic assertions, and so on.
  • DNF Disjunctive Normal Form
  • CNF Conjunctive Normal Form
  • the representation format can also be implemented directly as a hardware circuit, which may be implemented either using (i.) flexible architectures like FPGAs, or (ii.) more static architectures like ASICs or analogue/digital electronics, or (iii.) neuromorphic architectures that are suited for the hardware implementation of connectionist models such as XNNs, or (iv.) quantum computing hardware.
  • the representation transmission can be affected entirely in hardware when using flexible architectures that can configure themselves dynamically. Quantization and other compression techniques may be applied on the XNN for faster and more efficient execution.
  • the localized trigger is defined by a localization method, which determines which partition to activate.
  • a partition is a region in the data, which may be disjoint or overlapping.
  • the answer of a rule is a linear or non-linear equation which consists of coefficients with their respective dimension, representing both the answer to the problem and the explanation coefficients which may be used to generate domain specific explanations.
  • the XNN rules may also represent a hierarchy of rules (nested conditional rules or a series of nested if-then-else rules), that may represent any form of selection logic. For simplicity, examples are illustrated using a single hierarchical layer. However, the XNN can have multiple deep layers of nested rules in its architecture.
  • the nested rules may be converted into a logically equivalent single layer, for example one consisting of rules in disjunctive normal form (DNF) or some other equivalent.
  • DNF disjunctive normal form
  • Figure 4 may be a schematic flowchart illustrating rule-based knowledge or logically equivalent knowledge embedded in XNN.
  • a partition condition 402 may be chosen using a localization method that may reference a number of rules and encoded knowledge.
  • Partitions can be non-overlapping or overlapping.
  • the XNN may take a single path in feed forward mode.
  • the XNN may take multiple paths in feed forward mode and may compute a probability or ranking score for each path.
  • overlapping partitions may also make use of an aggregation function which combines the results from the activated partitions.
  • the partition condition 402 can be interpreted as focusing the XNN onto a specific area of the model that is represented.
  • the partition localization method may be typically implemented in the form given by template 404 where various features 406 are compared to real numbers 408 repetitively using conjunctive normal form (CNF) or a logical equivalent such as disjunctive normal form (DNF).
  • CNF conjunctive normal form
  • DNF disjunctive normal form
  • other non-Boolean logical systems may be utilized such as Type 1 or Type 2 fuzzy logic systems, modal logic, quantum logic, probabilistic logic or other suitable type of logical system for the expression of logical or similar statements.
  • the localization method values, conditions and underlying equations may be selected and identified using an external process, such as an XAI model induction method or a logically equivalent method.
  • the localization values, conditions and underlying equations may be partially or fully induced using an end-to-end approach using gradient descent methods such as back-propagation.
  • the chosen values are iteratively tested and varied until a certain level of accuracy is obtained in fitting the model to the relative test or synthetic data provided and/or by iteratively querying the initial black-box predictor model.
  • An XNN may have four main components in its localization or focusing module, which may be part of the conditional network 510, namely the input layer 410, a conditional layer 412, a value layer 414 and an output layer 416.
  • the input layer 410 is structured to receive the various features that need to be processed by the XNN.
  • the input layer 410 feeds the processed features through a conditional layer 412, where each activation switches on a group of neurons.
  • the conditional layer may require a condition to be met before passing along an output.
  • the input may be additionally analyzed by a value layer 414.
  • the value of the output X (in case of a calculation of an integer or real value, etc.) or the class (in case of a classification application, etc.) X is given by an equation X.e that is calculated by the value layer 414.
  • the X.e function results may be used to produce the output 416. It may be contemplated that the conditional layer and the value layer may occur in any order, or simultaneously.
  • Figure 5 may illustrate a schematic diagram of an exemplary high-level XNN architecture.
  • An input layer 500 may be inputted, possibly simultaneously, into both a conditional network 510 and a prediction network 520.
  • the conditional network 510 may include a conditional layer 512, an aggregation layer 514, and a switch output layer (which outputs the conditional values) 516.
  • the prediction network 520 may include a feature generation and transformation 522, a fit layer 524, and a prediction output layer (value output) 526.
  • the layers may be analyzed by the selection and ranking layer 528 that may multiply the switch output by the value output, producing a ranked or aggregated output 530.
  • the explanations and answers may be concurrently calculated by the XNN by the conditional network and the prediction network.
  • the selection and ranking layer 528 may ensure that the answers and explanations are correctly matched, ranked, aggregated, and scored appropriately before being sent to the output 530.
  • conditional network 510 and the prediction network 520 are contemplated to be in any order.
  • some of the components of the conditional network 510 like components 512, 514 and 516 may be optional or replaced with a trivial implementation.
  • some of the components of the prediction network 520 such as components 522, 524 and 526 may be optional and may also be further merged, split or replaced with a trivial implementation.
  • Figure 12 shows another exemplary embodiment where an additional feature generation and transformation component 509 may also be applied after the input layer and prior to the conditional/prediction networks. It may be further contemplated that an additional transformation step 511 may also be applicable within the conditional network. An additional transformation step 527 may optionally be applicable after the prediction 526. Step 527 may be required to apply a suitable activation function and/or suitable transformation on the predicted output. Typical activation functions may include the sigmoid or softmax functions or suitable equivalent.
  • Typical transformations may involve transformation of the prediction output using some form of structure such as (i) hierarchical tree or network, (ii) causal diagrams, (iii) directed and undirected graphs, (iv) multimedia structures, (v) sets of hyperlinked graphs, or suitable structures.
  • structure such as (i) hierarchical tree or network, (ii) causal diagrams, (iii) directed and undirected graphs, (iv) multimedia structures, (v) sets of hyperlinked graphs, or suitable structures.
  • the ranking layer 528 may also select, merge or split data from multiple partitions as shown in Figure 12.
  • the ranking layer combines the switch output with the prediction network output.
  • the ranking function may simply choose the value for the activated partition. It is contemplated that in a typical embodiment, any suitable transformation, selection, merge or split function may be applied within the ranking function
  • the selection and ranking layer 528 and the output 530 may be combined together into one integrated component.
  • the XNN may also be implemented with both the conditional network 510 and the prediction network 520 together with all their components merged together into one network. This merged conditional and prediction network may also be merged together with a combined selection and ranking layer 528 and the output 530. This optimization will still result in a logically equivalent XNN, which may be faster for feed forward processing.
  • Figure 16a and Figure 16b show exemplary embodiments of how the conditional network 510 may be combined with the prediction network 520 as one XNN network.
  • the partitioning system models the hierarchical concepts of the underlying dataset. It is contemplated that such a hierarchy of concepts may be linked to a suitable taxonomy and/or ontology, such as YAGO-SUMO, WordNet and other suitable taxonomies and ontologies.
  • Hierarchical concepts may be applied either by: (i.) utilizing the results of the leaf nodes only as depicted in Figure 16a with the hierarchy 800, or (ii.) by utilizing the results from all nodes as depicted in Figure 16b using the hierarchy 810.
  • hierarchy 810 it may be noted that the parent nodes Ri, R2 and R3 have been added to the ranking layer 528.
  • the ranking layer may also rank partitions such that their activation is either soft or hard, thus allowing the hierarchical partitions to model probabilistic or deterministic concepts.
  • the result 530 may also contain an activation function such as softmax or sigmoid.
  • Such embodiment may be trained using standard gradient descent techniques such as back- propagation. Standard loss functions such as categorical or binary cross entropy may be applicable, however, custom loss functions may also be incorporated to achieve further optimizations.
  • Figure 17 shows an exemplary embodiment 5f an XNN which makes use of a taxonomy for modelling the hierarchical partitions or hierarchical concepts.
  • the example depicted in Figure 17 makes use of the YAGO-SUMO taxonomy, however, any other taxonomy, ontology or knowledge graph may be used for modelling the hierarchical concepts or partitions.
  • Taxonomy links allow for building an explainable neural network (XNN) whereby the embedded knowledge mimic the structure as defined by a taxonomy. Human knowledge injection may also be incorporated to allow for custom links or adjustment which may be applicable for a specific use case.
  • the XNN in Figure 17 works by taking some input 750.
  • Such input may be the result of a deep neural network, such as a CNN, which could have been used for modelling the abstract concepts from raw input pixels.
  • the input 750 is then passed to the combined conditional/prediction network 760.
  • Such network is then used for finding the hierarchical weights, in a similar way to the XNN as depicted in Figure 16a or Figure 16b.
  • the results for the hierarchy are them combined together in the ranking layer 770 of the XNN, which is responsible for selecting, merging or splitting the activations from all the hierarchical nodes. In a trivial implementation 770 may be simply an activation or completely omitted.
  • the final result is then sent to layer 780 which may be used for classification or prediction purposes, or for integration with larger deep neural networks, including other explainable neural networks (XNNs).
  • XNNs explainable neural networks
  • the XNN can thus be implemented in a way that there is the input layer 500, and a combination of the conditional network 510 and the prediction network 520, including the conditional layer 512, aggregation layer 514, switch output layer 516, feature generation and transformation layer 522, fit layer 524, prediction layer 526, and ranking layer 528 leading to the output 530.
  • This combination may apply to all embodiments and implementations of the XNN, including both software and hardware implementations.
  • XNNs transformation capabilities of XNNs in this regard are unique and unparalleled in other neural network implementations since the white- box nature of XNNs allows flexibility and extreme merging to be performed without affecting the logical behavior of the XNN, although this affects various attributes of a practical implementation, such as size/space usage, performance, resource usage, trainability and overall throughput. It may be further contemplated that a hybrid neural network and transducer implementation, or logically equivalent white-box alternative, be used to combine parts of the XNN components without loss of functionality.
  • the XNN architecture as shown in Figure 12 may be implemented using fewer components. For instance, any of the components 509, 511, 512, 514, 516, 522, 524, 526, 527, 528 and 530 may be omitted, merged or split for optimization or implementation purposes.
  • the input 500 may contain data which has already been transformed, pre-processed or partitioned, thus eliminating some components in the XNN.
  • the output 530 may contain further transformations which are not visible in the general XNN architecture. Exemplary transformations may include, but are not limited to de-convolutions, un-pooling, time-series components, decoding, and so on.
  • Linear methods such as linear regression may predict a target as a weighted sum of input feature variables.
  • the linearity of the learned relationship facilitates interpretation.
  • y is the target variable and X 1 . . X m are the feature variables.
  • ⁇ 0 is the intercept, and ⁇ 1 .. ⁇ m determine the importance (i.e. weight) of each variable X towards the target y.
  • the target variable is a discrete value, which is why it is used for classification.
  • the interpretation of the weights in logistic regression differs from the interpretation of the weights in linear regression, since the outcome in logistic regression is a probability between 0 and 1, and the weights do not influence the probability linearly any longer beyond 1.
  • the weighted sum is transformed by the logistic function to a probability.
  • Figure 6 illustrates a schematic flowchart of a linear regression represented using a neural network.
  • Variables 602 may contribute to the output 604.
  • Various weights 606 may increase or decrease the impact of each variable 602.
  • the intercept 608 may also have an impact on the output, even though no variable is associated with the intercept
  • Figure 7 may illustrate a schematic flowchart of a logistic regression represented using a neural network.
  • the logistic regression neural network may implement variables 602 which may contribute to an output 604 and may be associated with various weights 606.
  • softmax is typically used as the activation function.
  • Activation functions are not limited to sigmoid or softmax. Any suitable activation function can be utilized to transform the output such as ReLU, and other functions.
  • linear methods are highly intuitive such as linear regression and logistic regression, they generally lack predictive performance for real world problems as they are only capable of modelling linear data.
  • Some workarounds such as polynomial expansions can be applied to model non-linear data, but deep neural networks generally provide higher predictive performance on complex non-linear relationships.
  • Figure 8 may illustrate the a linear fit and a polynomial fit on a scatter plot of random data.
  • the linear fit line 802 may be a straight line going substantially across an average of the data, whereas the polynomial fit line 804 may curve along with the data in order to better represent the set of non-linear data.
  • the resulting model may be intuitive.
  • modelling non-linearity simply involves adding multiple layers together with activation functions. This may be feasible, but the resulting model is a blackbox and may not be explainable in an intuitive way.
  • the XNN transform function may be a pipeline of transformations, including but not limited to polynomial expansions, rotations, dimensional and dimensionless scaling, Fourier transforms, Walsh functions, state-space and phase-space transforms, Haar and non-Haar wavelets, generalized L2 functions, fractal-based transforms, Hadamard transforms, Type 1 and Type 2 fuzzy logic, knowledge graph networks, categorical encoding, difference analysis and normalization/standardization of data.
  • the transform function pipeline may further contain transforms that analyse sequences of data that are ordered according to the value of one or more variables, including temporally ordered data sequences. [0095] Referring now to Figure 9, Figure 9 may illustrate an exemplary embodiment of a prediction network linear model 520.
  • the linear model 520 may be represented by or mathematically equivalent to the following exemplary polynomial equation: [0096]
  • An exemplary neural network as shown in Figure 9, is mathematically equivalent to the polynomial function f(x, y).
  • An exemplary input layer 500 may be used.
  • the XNN may include a feature generation and transformation layer 522 that contains polynomial transformation neurons as well as a concatenation or fit layer 524, whereby it may combine all polynomial features into one layer.
  • the feature generation and transformation layer 522 may produce a set of variables which are of relevance to the output.
  • the fit layer 524 may produce the set of coefficients (bo - b5 in the example shown in Figure 9) which weigh the variables obtained in the feature generation and transformation layer 522.
  • the coefficients may be analyzed, and a value may be produced in the value output layer 526.
  • the feature generation and transformation layer 522 is used to implement any advanced transformations that may be applied to the input features and relevant data, such as non-linear transformations.
  • polynomial transformation neurons may be utilized, which may model polynomial equations using a system of specially interconnected weights and neurons that enable a white -box approach to polynomial transformations.
  • the fit layer 524 may be used to implement any model fitting and prediction methods including but not limited to linear and non-linear fitting methods, Fourier series, multi-dimensional Bezier curves, recurrence relations, causal operators, etc.
  • the value output layer 526 combines the value answers together and ensures that they can be combined into a matched pair consisting of an answer together with its corresponding explanation.
  • Multiple such pairs may be generated if more than one partition has been selected by the XNN focusing step as determined by the conditional network 510.
  • the XNN can be implemented to work completely in parallel, generating plausible answer and explanation pairs which are then filtered and selected at a later stage via the output of the conditional network 510. Both parallel and sequential implementations of the XNN are thus possible, as the implementation can be tweaked to the particular application domain and also to fit within the implementation constraints of the software or hardware system being utilized.
  • High-degree polynomials may successfully be used in order to model complex data, yet may, however, still be uninterpre table, as well as computationally intensive in a high dimensional space.
  • An XNN may instead split the fit of the entire model into multiple smaller fits. This may be done through the concept of “partitions”.
  • a partition models a specific region within the entire model. Partitions can be overlapping or non-overlapping, depending on their configuration. In the case of overlapping partitions, some priority function may be used to determine which partition to activate, or alternatively, an aggregation function may be used to combine results from multiple partitions. The process to identify partitions may be part of the initialization process of the XNNs.
  • partitions may be done using either an external or integrated process which may utilize deep learning in an end-to-end or partial manner.
  • Standard loss functions such as mean squared error and binary/categorical cross entropy are applicable to XNNs. It may be further contemplated that a custom loss function may also be applied to XNNs such that it minimizes overlaps between partitions and minimizes the error between the prediction and the labeled output.
  • Hierarchical partitioning techniques or other logically equivalent methods can be used for identifying suitable partitions, such as an XAI model induction method which may input a set of training data to a black-box system, and analyze the corresponding output using a partitioning function.
  • the partitions may be initially created as part of the XNN by an external process that creates the initial XNN, such the previously described model induction method.
  • partitions may also be pre-trained or suitably initialized from a linked taxonomy or ontology that provides suitable partitioning information.
  • the XNN can use gradient descent methods to fit or further refine the partitions if desired.
  • the XNN partitions may either be locked by having their values set as static values that cannot be changed, or can be movable dynamically for each backward training pass using standard back-propagation and related techniques.
  • the hierarchical partitions may model data at different levels of abstraction. Such abstraction may be required when generating explanations.
  • Figure 13 shows an exemplary embodiment whereby only the leaf nodes/partitions are used for the purpose of ranking partitions in step 528.
  • all nodes in the hierarchy may be used within the ranking layer 528.
  • both XNNs in Figure 13 and Figure 14 may be logically equivalent, depending on the implementation of the ranking function 528, which determines how to select, merge or split the overlapping or non-overlapping partitions.
  • the hierarchical nature of the XNN partitions may be used to represent symbols and concepts at different levels of semantic and semiotic detail.
  • XNN partitions may be hierarchical rather than just a flat partition.
  • flat partitions may be encompassed by the definition of a hierarchical structure. As such, flat partitions may be supported as a sub-set of hierarchical partitions where there is just a single level.
  • the utilization of a hierarchy allows for more expressive explanations to be generated that more adequately models the expected nature of a good explanation.
  • DARPA XAI Program literature may be used as an example of an objective guideline for the features of a good explanation.
  • an explainable AI system may comply with the desired goals of the XAI concept if the user can answer the questions “why did you do that?,” “why not something else?,” “when do you succeed?,” “when do you fail?,” “when can I trust you?,” and “how do I correct an error?”.
  • XAI contemplates that clarity of the explanation and utility of the explanation may be balanced against one another, with explanations that are too superficial for detailed analysis and explanations that are too detailed and complex for a user to make a useful decision based on the explanation both being ineffective from a user satisfaction standpoint. Since appropriate levels of clarity and utility may vary based on the user and based on the intended application, it may readily be contemplated that a hierarchical, multi-level explanation that can selectively hide or show parts of an explanation or dynamically expand or shrink it is a superior type of explanation to a simple flat level explanation. Conformity to other standards other than the XAI guidelines may likewise be contemplated, such as may be desired.
  • XNNs ensure that the best practices in explanation encoding, transmission and eventual conversion to a human-readable or machine -readable higher- level domain specific explanation are fully supported.
  • Multiple hierarchical partition techniques exist that may be utilized for XNN initialization including hierarchical partitioning, lazy clustering, agglomerative clustering, divisive clustering, relocation partitioning, probabilistic clustering, k-mediod methods, k-means methods, fuzzy clustering, density based clustering, grid based methods, gradient descent based methods, evolutionary methods, region splitting, region growing, sub-space clustering, projection methods, co-clustering methods, lazy clustering methods, etc.
  • the number of partitions generally depends on the overall complexity of the original predictor model or the underlying dataset - the more complex, the more partitions one typically creates. Multiple partitions may be connected with a “switch”. The switch may determine which partitions to activate through the binary values. The resulting switch is then multiplied with the linear output.
  • XNNs may be designed to work with different non-linear transformations, including but not limited to polynomial expansion, Fourier transforms, continuous data bucketization, causal operators, and so on. Conditional features may also be applied. XNNs can also apply intelligent merging and summarization of partitions at multiple levels of detail, while also catering to some scale and rotational invariances whenever possible. Adjacent partitions in multi-dimensional space that have a possible fit in multi-dimensional space which summarize one or more partitions more efficiently will get an associated summary that may be more amenable to explanation generation. [0105] Scale invariance may be accounted for by appropriate coefficients which adjust the scale in multiple dimensions.
  • Rotational invariance is accounted for by having appropriate rotational coefficients that adjust rotation in multiple dimensions. This is a powerful feature of XNNs that enables non-linear data to be explained in a superior manner than methods that simply use linear methods exclusively. For example, if multiple partitions representing a rotated elliptical boundary exist in the XNN model, these partitions can be merged into a single representation of a rotated ellipse, giving a precise yet highly summarized representation of potentially complex boundaries that may be voluminous to describe using linear fitting. The resulting summary explanation is also more accurate than could be achieved using linear fitting and also follows the widely accepted principle that a shorter explanation is preferred over a longer explanation whenever available.
  • Figure 10 may illustrate a schematic diagram of a conditional network 510.
  • the input 500 of the conditional network 510 may be the same input as used in the linear model 100.
  • the conditional network may be eliminated thus resulting in a model with just one global partition.
  • the hierarchical partitions or clusters may be integrated within the prediction network, thus having a hierarchical structure of predictions.
  • the exemplary schematic diagram of a conditional network 510 is an expanded possible implementation of the conditional network 510 shown in Figure 5.
  • the conditional network 510 may include three main layers that can be combined together depending upon the implementation constraints. The first of these may be the conditional layer 512, which is initialized according to the hierarchy of partitions being implemented in the XNN.
  • the second of these may be the aggregation layer 514, which is initialized for the first time by the external process that defines the XNN and can be subsequently refined using backward pass training methods such as gradient descent methods, etc.
  • the third of the main layers may be the switch output layer 516 that combines the results of the aggregation layer 514 and conditional layer 512 together.
  • the switch output layer 516 is also initialized by the same external process that defines the XNN and is typically implemented as the same process that initializes the aggregation layer 514, although it may be contemplated that these processes may be independent if necessary. All values in the connection weights between the neurons can be updateable using standard neural network training techniques. XNNs provide a further novel method of updating the connections themselves using machine learning methods such as genetic algorithms, Monte Carlo simulation methods, simulated annealing, reinforcement learning, etc. or via causal logic models.
  • Each condition may be split into partitions.
  • Each box in the conditional layer 512 may represent a single partition.
  • “y > 15” may be a single partition, representing a single rule which applies where “y > 15” (and, thus, not in the alternative case where y ⁇ 15).
  • the partition may be combined with another partition in the aggregation layer 514.
  • the partition “y > 15” is combined with the partition “x > 20”.
  • the result may be found following the activation function 1120.
  • the results combination may be obtained via a suitable transformation function or activation function.
  • the transformation function or activation function may be applied either: (i.) on the predicted value as shown in step
  • 11 may begin with an input 500.
  • the input may then be used as inputs to the conditional network
  • the prediction network may contain a feature generation and transformation layer 522, a fit layer 524, and a value output layer
  • the value output layer 526 may provide the result of the equations which correspond to rules which weigh different features of the inputs.
  • the result 526 is combined with the switch output
  • the combined result 1116 may contain results from one or more partitions.
  • One way to combine the results 1116 is to utilize element-wise multiplication. Some ranking function or merge function may then be used to calculate the result 1118.
  • the input 500 may be used as input to the conditional network 510, as illustrated in Figure 5 and Figure 10. Again, the conditional layer 512 and aggregation layer 514 may produce conjunctive rules or other logical equivalents or partitions which are represented in the switch output layer 516. [0112]
  • the outputs of the value output layer 526 and the switch output layer 516 may be combined in the output layer 530. Once the output layer 530 has been formed, a sigmoid or other activation function 1120 may be applied to the result 1118, depending on the application.
  • the prediction network 520 may incorporate hierarchical concepts when combined with the conditional network 510.
  • the image or concept of a dog may be the input 700 to the AI system.
  • concepts may be extracted with a deep neural network such as a CNN network.
  • the AI system may use an image classification algorithm.
  • the AI system may classify the input image by applying a set of rules to the image and determining whether the rules are satisfied or not.
  • the rules may be implemented in a hierarchical structure, although a linear or other structure may also be utilized in a practical implementation.
  • An exemplary embodiment may first analyze a portion of the dog, such as the legs 702.
  • R 1 may be a rule which is triggered when an object with 4 legs is detected. Since R 1 was triggered by the input image, the system may then be directed to apply rules R 2 and R 3 to the input image.
  • Rule R 2 may be a conditional rule which may be triggered by the appearance of a tail 704. The rule may be stored or formulated in the disjunctive normal form (DNF). Rule R 2 may be triggered by conditions such as a thin object which protrudes from another, thicker object of the same or a similar color (the body) which also satisfy R 1 (the rule which identified the legs). Once rule R 2 has been satisfied, the AI system may be directed to rules R 4 and R 5 . Rule R 4 may identify the snout 706 of the dog.
  • DNF disjunctive normal form
  • Rule R 5 may be an additional rule which is triggered by the presence of a collar 708. Rule R 5 may not be necessary for the classification, however, R 5 may increase the chances that an accurate classification was made.
  • the XNN network may utilize gradient descent techniques such as back-propagation to find the values of R 1 to R 7 .
  • the resulting values of R 1 to R 7 may act as probabilistic values such that the resulting classification is the argmax of R 4 , R 5 , R 6 and R 7 .
  • XNNs may present an intuitive way to construct interpretable models, while still utilizing the power of ANNs and related methods such as deep learning. Once the model is trained through back-propagation or a similar method, the resulting neural network can be used to serve predictions and the inner structure of the XNN can be used to construct the rules.
  • the resulting explanation from the XNN can be represented using a universal explanation format.
  • the explanation can then be processed further to give a human readable explanation that is suitable for whatever domain application the XNN is being used for.
  • XNNs can form a single module or layer within a larger neural net - as long as the explanation outputs are preserved, the resulting network will also be a white -box model. If the explanations are not carried over and preserved within the network, then the resulting model may no longer be a pure white -box.
  • Output from a black-box AI system can be used as input to an XNN, such as one that has been created by a model induction method, without changing anything.
  • Output from the XNN model can be used as input to one or more XNNs, or to another AI system that is receiving the XNN output.
  • the receiving AI system is a white -box system, this preserves the white-box properties of the combined system. If the receiving AI system is a black- box system, the system may be partially black-box. Since XNNs can preserve explainability data even when chained, various implementations may be possible.
  • XNNs can also be transformed to optimize their architecture for speed, performance, trainability, resource usage or a combination of these or other different factors.
  • white-box nature of XNNs allows such logical transformations to be applied, something that is impossible to do with standard ANNs and Deep Learning methods.
  • XNNs may be compatible with a universal format for explainable AI (XAI) explanation encoding and transmission.
  • XNNs may also be compatible with explainable technologies, such as induced XAI models, interpretable neural networks (INNs), explainable transducer transformers (XTTs), explainable spiking nets (XSN), explainable memory nets (XMN), and the like, as well as reinforcement learning applications where they may be compatible with explainable reinforcement learning (XRL).
  • explainable technologies such as induced XAI models, interpretable neural networks (INNs), explainable transducer transformers (XTTs), explainable spiking nets (XSN), explainable memory nets (XMN), and the like, as well as reinforcement learning applications where they may be compatible with explainable reinforcement learning (XRL).
  • an XNN may be a type of white-box Wide Neural Networks (WNNs). WNNs complement the mainstream black-box Deep Neural Networks (DNNs) and provide significant advantages over DNNs. Independent analysis of WNNs in (Zagoruyko and Komadikis, 2017) shows that WNNs may significantly outperform logically equivalent yet much more structurally complex DNNs by magnitudes of order, and also WNNs may be several times faster to train than DNNs. XNNs may provide both these two additional advantages, and our initial experimental results from XNN implementations strongly indicate that these advantages are applicable to most XNNs.
  • WNNs white-box Wide Neural Networks
  • XNN nodes can also be linked to a taxonomy system to achieve taxonomical links and knowledge to be seamlessly incorporated and added at various steps in the process, to achieve fusion of human and machine knowledge while providing a link to causal logic and further generalization and induction of new knowledge.
  • XNNs can also be used for bias detection in the input data and/or the resulting model.
  • the global nature of XNNs can also be used to detect areas of strength and weakness in the underlying data training set. XNNs can thus allow interactive and directed feedback from the AI system to humans in the collection of future training data by identifying gaps and weaknesses that need to be addressed.
  • CNN-XNNs allow convolutional layers to be seamlessly integrated with XNNs, giving them the capability of processing images, 3D data and other signal processing that is amenable to convolutional feature identification, transformation and extraction.
  • CNN- XNNs may incorporate a set of one or more convolutional layers as part of a pre-processing step between the input features and the input to the conditional and prediction networks of an XNN.
  • the original features together with the output from the convolutional layers are both treated as input to the XNN.
  • the convolved features may also be used in conditions placed in the conditional layer.
  • the output from the convolutional layers may also be dimensionally reduced and also potentially partially or fully deconvolved.
  • the convolutional filters or kernels may be used in the generation of an appropriate explanation in an intuitive and human readable manner.
  • Example applications of CNN-XNNs involve image interpretation and diagnosis explanation of X-Ray and MRI images in medical devices; LIDAR and stereoscopic image data explanation and fused vision-control model explanation in autonomous air, land, sea, underwater and space indoor and outdoor vehicles; object and event detection and explanation of images in various application domains such as traffic cameras, UAV and satellite imagery, social media network photos, etc.
  • Most currently known image or video or 3D data-based applications of Artificial Intelligence can incorporate CNN-XNNs to add explanatory capabilities to such applications.
  • Capsule XNNs preserve hierarchical information captured during the convolution steps by a CNN-XNN. CAP-XNNs may thus be considered as an alternative variant to standard CNN-XNNs.
  • CAP-XNNs add a capsule network in between the input layer and the condition and prediction layers in the XNN.
  • CAP-XNNs are similar to CNN-XNNs but since the capsule network can also preserve hierarchical information, and XNNs utilize hierarchical partitions and hierarchical notions of explanations, such hierarchical information from the capsule network layers can be directly used by the XNNs.
  • CAP-XNNs may thus offer different performance than CNN-XNNs due to the addition of hierarchical information.
  • the example applications of CAP-XNNs are identical to CNN-XNNs.
  • CNN-XNNs and CAP-XNNs are largely completely replaceable by each other, ensuring that a CNN-XNN can be swapped in for a CAP- XNN and vice-versa.
  • the CAP-XNN combination is likely to be more computationally powerful and expressive due to the presence of hierarchical information.
  • Alternative current and future enhancements to CNNs that add hierarchical information will also be compatible with XNNs, opening up possibilities of having other XNN options that are not based on capsule networks but some other variant on convolutional networks. As long as the convolutional aspect is kept, these variations may be implemented into an XNN.
  • Text XNNs are a variant of XNNs that can handle text processing and textual data including syntactic and semantic information.
  • T-XNNs may include a processing step for the input features that transform text data into suitable vector representations, and may thus incorporate techniques like word2vec and end-to-end processing for textual feature engineering, and the like.
  • T-XNNs may typically take advantage of the white -box nature of XNNs to optionally incorporate taxonomical knowledge and also external linguistic knowledge as part of both the conditional and prediction networks in the XNN. The incorporation of such knowledge may lead to an improvement in the T-XNN performance and also in its explanatory expressiveness.
  • T-XNNs may be typically combined with sequence processing in the XNN prediction network, such as the extensions described for Predictive and Recurrent XNNs (PR-XNNs).
  • Example applications of T-XNNs are as part of a document understanding system; a machine document translation system; an information retrieval or other form of search engine system; a semantically based knowledge extraction and understanding system, as well as any other embodiments that have to do with text processing.
  • T-XNNs also open up the possibilities of having better control and understandability of the resulting neural network models, which is a barrier to further improvement in traditional black-box neural networks.
  • a T-XNN can explain the learnt grammatical rules for a particular set of training documents, corpus, or input language, which can then be analyzed and improved collaboratively using a human review step.
  • the initial grammatical rules may be learnt automatically using the T-XNN initialization process without human intervention. From the first review onwards, such reviews can then re-incorporate suggested edits into the T-XNN, allowing for a virtuous cycle of machine and human collaboration and semi-supervised learning to improve performance.
  • T-XNNs support supervised, semi- supervised and unsupervised modes of learning. As T-XNNs are also still compatible with gradient descent and other neural network training methods, all the advantages of re-training are available for a T-XNN.
  • Speech XNNs are an extension of XNNs for speech recognition and generation, adding an explanation to speech recognition by inserting an XNN or alternatively using an induced model as part of the speech recognition solution.
  • S-XNNs incorporate a speech model as part of a pre-processing step between the input features and the input to the conditional and prediction networks of an XNN.
  • the original features together with the output from the speech model are both treated as input to the XNN.
  • the speech model features may also be used in conditions placed in the conditional layer.
  • the output from the speech model may also be dimensionally reduced and also potentially partially or fully encoded or decoded.
  • the speech model data itself may be used in the generation of an appropriate explanation in an intuitive and human readable manner.
  • Speech models may be in the form of a speech specific neural net or other form of machine learning model such as Bayesian networks, HMMs, and other models used in automated speech modelling.
  • an S-XNN can, for example, be used to generate explanations of why a particular phoneme and intonation has been selected.
  • S-XNNs can also be used to explain why particular interjections, fillers, non-lexical vocables and other breaks and irregularities have been inserted in the speech output to make them sound more natural.
  • Example applications of S-XNNs are for automated speech recognition systems; automated speech generation systems; intelligent personal, home and office assistants; speech based control software and hardware systems, like those used to control a wide variety of industrial machinery and in transportation and aviation; speech control systems for pilots and future autonomous vehicle applications; speech based interfaces and agents; call center and telemarketing systems; telecoms hardware that utilizes automated speech recognition and generation; conversational and dialogue agent systems and the like.
  • Dialogue and Conversational XNNs automate conversational, dialogue and question & answering (Q&A) systems in combination with XNNs. Given a specific question, an explainable neural network outputs the answer along with an explanation as to why such answer has been specified.
  • QA-XNNs incorporate additional context in the conditional and prediction networks of an XNN to keep track of the current status of a conversation, dialogue or Q&A session.
  • QA-XNNs may also incorporate multiple XNNs within an overall control loop structure or system that enables multiple iterative requests and queries to be executed over the duration of a user session.
  • the XNN explanation coefficients for a QA-XNN incorporate state information and if used in an overall control loop system, some form of positional information to help generate an appropriate response to that particular loop or iteration in a session.
  • QA-XNNs take advantage of the white -box nature of XNNs to incorporate additional context sensitive information especially in the prediction network in a seamless manner. It is contemplated that QA-XNNs will be used to replace existing neural networks in existing Q&A systems in a transparent manner.
  • QA-XNNs will be combined with Predictive and Recurrent XNNs (PR-XNNs) and Causal XNNs (C-XNNs) in a number of practical embodiments.
  • PR-XNNs Predictive and Recurrent XNNs
  • C-XNNs Causal XNNs
  • QA-XNNs will also be used in conjunction with T-XNNs, unless the input features for the QA-XNNs have already been processed by some other separate system that has encoded semantic attributes and knowledge, etc.
  • Example applications of QA-XNNs include an automated chatbot system such as those used for automated reservations and assistance; interactive expert systems and search engine systems; and in applications where the reason underlying a particular answer needs to be further explained.
  • QA-XNN The advantage of using a QA-XNN is the addition of the capability of the implementation to be capable of answering questions that refer to why a particular answer has been given by the system, allowing for better interaction between people and machines and also enabling a better understanding to form as a direct consequence. Feedback from this process can also be used in subsequent re- training and refinement processes to make Q&A systems that utilize QA-XNNs to improve their accuracy and usefulness at a faster accelerated rate than other systems that do not utilize explanations as part of their improvement and training pipeline.
  • PR-XNNs Predictive and Recurrent XNNs add an element of time and or sequence to the input and to the output.
  • Such XNNs can match an input sequence and identify its trend while outputting and forecasting possible outputs together with the explanation for each possible output.
  • Such XNNs may utilize recurrence in the explanation and have parts of the explanation refer to itself in a modular manner.
  • the specific type of predictive architecture may give rise to different variants of PR-XNNs, for example a long short-term memory unit (LSTM) PR-XNN or a gated recurrent unit (GRU) PR-XNN.
  • LSTM long short-term memory unit
  • GRU gated recurrent unit
  • PR-XNNs add contextual positional information in the conditional and prediction networks of an XNN. This contextual positional information may also form part of the output if required, depending upon the specific application and implementation. The contextual positional information may also be added to the matched answer and explanation pairs, as it may be necessary to precisely distinguish PR-XNN output as part of a sequence of output items rather than a single output item.
  • PR-XNNs are contemplated to typically have some form of recurrence relations in the prediction network, and to a lesser extent may also need recurrence relations in the conditional network.
  • the recurrence relations are learnt during the PR-XNN initialization step from the original predictor model. For example, if the original predictor model is an LSTM network, the recurrence relations may reflect the LSTM behavior as part of the underlying explanation given by the PR-XNNs.
  • the LSTM model may be implemented as part of the prediction network of the PR-XNN; or alternatively, the original predictor may be placed either before or after the input layer of an XNN.
  • the resulting PR-XNN will treat the output of the original predictor as a sequence of input features, just like any other black-box predictor model that is induced to a white-box XNN.
  • the original predictor model is placed after the input layer of an XNN, the resulting PR- XNN will have an extra data that ensures that any explanation coefficients are passed through the predictor model.
  • an LSTM is placed in between the input layer and the conditional and prediction networks of an XNN, or an LSTM is incorporated as part of the prediction network of an XNN, the LSTM model needs to pass through the explanation coefficients in order to make sure that the resulting PR-XNN is still a white -box.
  • This passthrough innovation allows a PR-XNN to remain a white-box even if an element in its prediction is effectively a black-box or a grey-box with a mix of white -box elements and black-box structure.
  • PR-XNNs can also incorporate an LSTM or similar model after their output layer, allowing the LSTM to learn to predict the explanation itself. It may be contemplated that similar models, such as RNNs, Transformer, Bayesian Networks, Markov Models and other suitable models may be utilized instead of an LSTM. Such a combination of LSTMs or other similar models with PR- XNNs can lead to efficient new systems that not only generate an explanation from a single data item and/or ordered or unordered sequence of data items, but also be capable of predicting possible explanations. This capability of PR-XNNs makes them particularly suitable for industrial applications that need to anticipate the consequences of planned actions prior to actually executing them.
  • PR-XNNs can thus open up a new set of capabilities in industrial and commercial and non-commercial applications that allow safety to be part of the control loop of automated machinery, apparatus and systems.
  • PR-XNNs may allow practical systems to safely operate automated machinery based on the anticipation and prediction of consequences.
  • the ability to guarantee a safe mode of operation of machinery and robots, especially machinery and robots which interact with people, is a major unresolved problem which PR-XNNs may solve to a large extent.
  • Video XNNs are a combination of CNN-XNNs and PR-XNNs whereby a sequence of images (frames) are used to predict, anticipate and explain what is likely to happen in future frames. This combination may be common enough to warrant an explicit variant of XNNs to make implementation easier by re-using best practice implementations.
  • Video XNNs are not limited to 2D video.
  • a 3D data stream processed by the CNN-XNN will enable the XNNs to process and explain 3D data such as stereoscopic videos, LIDAR data, RADAR, SONAR, and the like.
  • V-XNNs incorporate a CNN-XNN generally prior to the input for a PR-XNN, thus having the CNN-XNN acting as a pre-processing step for the input to the PR-XNN.
  • the PR-XNN output may also be incorporated in a feedback loop that goes back to the CNN-XNN.
  • Sensor data such as 2D or 3D video, will generally be fed into the CNN-XNN.
  • alternative arrangements such as a PR-XNN followed by a CNN-XNN and a subsequent second PR-XNN can also be useful in a practical implementation of a V-XNN.
  • V-XNNs can also be used to process non-video data efficiently, for example, stock price information or time-series data as long as there is a defined sequence and ordering to the input information.
  • Example applications of V-XNNs would be in medical devices that perform continuous scans of a patient, such as during operations involving fluoroscopy equipment that constantly scans patients with X-Rays while a surgeon is operating.
  • V-XNNs in such a situation may provide quasi- realtime feedback to the surgeon of any diagnosis together with its explanation, which is important especially in a time critical situation like a surgical intervention.
  • Another example application of a V-XNN is during troubleshooting of industrial machinery, where scans are being constantly taken and anomalies need to be detected. Subsequent interventions and troubleshooting steps will force the system to adapt its answers and explanations to new situations, with the resulting explanations making V-XNNs a more suitable solution than other solutions that do not give an explanation.
  • V-XNNs may be combined with other XNN variants like a QA-XNN to implement interactive solutions that need to have situational awareness combined with interventions and actions that affect the physical world, enabling the system to adapt to changing circumstances while all the time maintaining its capability to explain itself. Combining such a system with a PR-XNN also gives it the capability to anticipate the future to some extent, giving it further useful capabilities that are beyond the scope of current black-box neural network- based systems.
  • XGANs explainable Generative Adversarial Networks
  • GANs Generative Adversarial Networks
  • XGANs utilize XNNs instead of standard neural networks giving rise to explainability for both the generator and the discriminator.
  • An XGAN utilizes an XNN in either the generator or discriminator or in both parts of a GAN system.
  • the advantage of an XGAN over a standard GAN is that the explanation information present in an XNN becomes available in a generative adversarial system, allowing the XGAN to have more complex behavior beyond what a GAN can do. For example, in an XGAN application that tries to detect anomalies efficiently over a telecoms network by generating potential examples of anomalous situations, the explanations can be used by the discriminator to distinguish between true and false alarms more efficiently than in the case when no explanations are available.
  • a reinforcement learning (RL) training method can utilize XNNs to create an Explainable Reinforcement Learning Model (XRL).
  • the XRL may include the generation of explanations in the action plan and world model components of the RL system.
  • the XRL may use the explanations themselves as part of the cost and reward functions.
  • XRLs can be extended to the entire class of techniques based on agent-environment interactions, for example those that are based on Markov decision processes, game theory and also partially observable Markov decision processes.
  • XRL may use the explanation information as part of the feedback, error, reward, cost, state space, action space, etc.
  • Example applications of XRL would be in robotics, where XRLs may utilize the explanation information to anticipate safety issues and minimize or prevent unsafe operation modes; in vehicle traffic control, XRLs can utilize explanations about the anticipated behavior of vehicles to ensure a better flow and throughput and anticipate potential risks of accidents and bottlenecks more efficiently while also being extensible to a mix of human and autonomous traffic; in resource logistics and planning, XRLs can utilize explanations about the actions of the various agents involved for example in a warehouse to optimize the behavior of autonomous systems such as autonomous forklifts.
  • Explanation information may be critical in such situations to avoid erroneous or potentially illogical actions being taken by such automated systems leading to errors that humans would typically never do, for example by trying to pack a box that has accidentally fallen onto the floor without attempting to pick it up again.
  • An XRL faced with such an explanation would modify the action plan to include a sequence of commands to re-attempt to pick up the box before proceeding with the rest of the plan, altering the cost analysis along the way to accurately reflect the change in the world situation.
  • C-XNNs Casual XNNs
  • C-XNNs extend XNNs to integrate with causal logic models giving the explanations the power of causality.
  • C-XNNs can incorporate casual logic models as part of the prediction network in an XNN.
  • XNNs and/or C-XNNs output may also be incorporated as part of the input in a causal logic model to form another variant of C-XNNs.
  • C- XNNs can also utilize a causal logic model to create explanations that take causality into effect, thus providing an explanation that goes beyond a simple description of a particular scenario, opening up a new range of explanations that give a chain of cause-and-effect scenarios and reasons.
  • C-XNNs are also capable of generating explanations that have a what-if but also a what-if-not nature.
  • a C-XNN can be used to generate a medical diagnosis for a particular condition but also explain the cause-and-effect of that diagnosis.
  • the resulting explanations can cover not just what the system has detected but also the reasons for exclusions of other potential symptoms or diagnoses.
  • a C-XNN can be combined with an XRL system to create systems that can anticipate and act in a physical (or simulated) world, explain themselves and also provide automatic input to the XRL simulation system that further improves the performance of the combined system as a whole.
  • a combined C-XNN and XRL system may be used in the previously mentioned warehouse application to detect that a particular type of box keeps falling from a forklift, then using the C-XNN to provide the right parameters to simulate the right amount of pressure increase or decrease or different change in the procedure commands for picking up such particular type of box, and then subsequently using the results of that simulation to update the XRL system.
  • XNNs remain compatible with the generic XNN architecture, meaning that they can be mixed and matched in various combinations as part of a larger and more sophisticated XNN.
  • a C-XNN can be combined with a CNN-XNN or a V-XNN to provide casual explanations for an autonomous vehicle.
  • Figure 18 shows an exemplary embodiment how XNNs may be combined with other AI systems, in this case an exemplary CNN-XNN system.
  • the implementation works by taking some raw input pixels 900, which are then transformed through a number of layers such as convolution and pooling layers.
  • the output of the CNN layers 910 becomes an input to the standard XNN architecture 920 by connecting 910 with the XNN original input 500.
  • the output of the XNN 530 may then be used for further processing or transformations.
  • the output 530 is connected with the output layer 930.
  • the output layer 930 may be used to transform the XNN output 530 further, for example, to make it compatible with some particular output device, or particular system utilizing said output, or as part of some appropriate transformation pipeline. In other exemplary embodiments, the output layer 930 may be used to connect the XNN within larger neural networks or suitable systems. In other exemplary embodiments, the output of the XNN 530 may be utilized by adding more layers and transformations including but not limited to de-convolutions, un-pooling, time-series components, decoders, and so on. In other exemplary embodiments, the output layer 930 may be omitted entirely and/or merged with the output of the XNN 530.
  • XNNs can also be used alone or in combination with other XNNs and variants of XNNs to utilize the resulting answers and their explanation(s) to act as a trigger(s) for a suggested action or set of actions in a general goal/plan/action system.
  • Some XNN variants such as PR-XNNs are also capable of outputting an ordered sequence of triggers and/or suggested actions, making them highly suitable for goal/plan/action systems, robotic systems, RL systems, etc.
  • an explainable neural network adapted to provide predictions and explanations
  • the explainable neural network comprising: an input layer configured for receiving inputs to the explainable neural network; a conditional network configured to receive said inputs from the input layer, wherein the conditional network is adapted to split said inputs into a plurality of partitions based on a set of rules; a prediction network configured to receive said inputs from the input layer, wherein the prediction network is adapted to fit at least one model associated with the plurality of partitions such that a combination of said at least one model forms a representative global model in relation to the plurality of partitions; a ranking layer configured to receive a combined output from the conditional network and prediction network; and an output layer configured to receive the combined output for providing the predictions and the explanations.
  • conditional network further comprising: a conditional layer configured to partition said inputs received from the input layer, wherein each partition of the plurality of partitions corresponds to at least one rule of the set of rules; an aggregation layer configured to combine outputs of the plurality of partitions with being presented as conditional values; and a switch output layer configured to output the conditional values as results pooled from the plurality of partitions.
  • the switch output layer is configured to identify and select the partitioned inputs for outputting as the conditional values.
  • said inputs are split hierarchically into partitions in the conditional network.
  • the prediction network layer further comprising: a feature generation and transformation layer configured to apply one or more transformations to said inputs received from the input layer, wherein the feature generation and transformation layer generates a set of variables based on the said inputs; a fit layer configured to model said inputs locally based on the set of variables generated from the feature generation and transformation layer, wherein the fit layer estimates coefficients corresponding to each variable of the generated set or a combination thereof in relation to said inputs; and a value output layer outputs a predicted set of outputs based on the coefficients estimated from in the fit layer.
  • the fit layer estimates coefficients corresponding to each variable of the generated set or a combination of the set of variables in relation to said inputs and at least one partition of the plurality of partitions.
  • said at least one model associated with the plurality of partitions establishes a link between each model and a partition of the plurality of partitions, wherein said at least one model may be a local model.
  • the explainable neural network further comprising: one or more transformation networks situated between the input layer and both the prediction network and the conditional network, wherein said one or more transformation networks is configured to transform said inputs before received respectively by the prediction network and the conditional network.
  • the prediction network further comprising: at least one transformation layer configured to transform said inputs before and/or after said inputs are received by the fit layer.
  • the conditional network further comprising: at least one transformation layer configured to transform said inputs before said inputs are received by the conditional layer and after received by the conditional network.
  • said at least one transformation layer comprises one or more transformations that are adapted to be applied to said inputs both linearly, non-linearly, or both.
  • said one or more transformations comprise a pipeline of transformation functions.
  • said one or more transformations comprise at least one transform that is adapted to analyze a sequence of said inputs, wherein the sequences is ordered in accordance with the set of variables associated with the feature generation and transformation layer.
  • the sequence is ordered temporally or as at least one temporally ordered sequence.
  • the explainable neural network further comprising: one or more modules, wherein said one or more modules comprise a normalization module configured to normalize said inputs before received by the input layer or after outputted from the input layer, a scaling module configured to scale the said inputs before received by the input layer or after outputted from the input layer, and a standardization module configured to standardize the said inputs before received by the input layer or after outputted from the input layer.
  • a normalization module configured to normalize said inputs before received by the input layer or after outputted from the input layer
  • a scaling module configured to scale the said inputs before received by the input layer or after outputted from the input layer
  • a standardization module configured to standardize the said inputs before received by the input layer or after outputted from the input layer.
  • the explainable neural network further comprising: at least one activation function situated in the prediction network or the output layer, wherein said at least one activation function is configured to activate or transform a portion of neurons in the respective network or layer, wherein the portion of neurons comprises one or more neurons.
  • the ranking layer combines the result from the prediction network with selected partitions from the conditional network via the switch layer.
  • said selected partitions comprise activated neurons.
  • the ranking layer selects from the plurality of partitions for subsequent activation when neurons associated with more than one partitions in the conditional network are activated.
  • the explainable neural network further comprising: an aggregation module configured to combine or further divide said outputs of the plurality of partitions of the conditional network and/or prediction network.
  • the explainable neural network is adapted to embed a plurality of artificial neural networks.
  • the conditional network and the prediction network simultaneously configured to receive said inputs from the input layer at same time.
  • the set of rules for partitioning said inputs comprises encoded knowledge adapted to localize the explainable neural network.
  • the set of rules comprises a rule for splitting at least a portion of the plurality of partitions, a rule for merging at least a portion of the plurality of partitions, and a combination of rules for splitting and merging the plurality of partitions; wherein the combination of split and merge the plurality of partitions according to one or more sequential patterns.
  • the explainable neural network further comprises a feedforward step traversing the explainable neural network across multiple paths, and a probability or ranking score is computed for each path of the multiple paths.
  • said at least one partition of the plurality of partitions overlaps with the second partition of the plurality of partitions, and an aggregation function is applied to combine the conditional values to be outputted.
  • the explainable neural network comprises a feed-forward step traversing the explainable neural network across multiple paths and a probability or ranking score is computed for each path of the multiple paths.
  • the explainable neural network is further configured to form the plurality of partitions based on comparing said inputs with a set of localization values and conditions, wherein the set of localization values and conditions are selected and identified using an internal and/or an external process in an inductive manner.
  • the internal and/or the external process comprising: a local and/or global optimization algorithm configured for implementing human knowledge and machine-generated knowledge into the plurality of partitions, wherein said human knowledge and machine-generated knowledge are associated with at least one type of: taxonomical information, internal and external taxonomies information, bias detection information, and knowledge bases and rule system generated information.
  • the set of localization values and conditions are induced using one or more end-to-end model or techniques adapted to optimize the explainable neural network.
  • the method comprising: receiving training data by an input layer of the explainable neural network, wherein the training data comprise at least one input and output pair; applying a gradient estimation algorithm to compute a gradient of a loss function with respect to weights of the explainable neural network for a pair input and output of the training data; and configuring the explainable neural network to update the weights with respect to the training data, wherein weights are updated using a deterministic or a heuristic optimization algorithm adapted to optimize the explainable neural network locally and globally.
  • said training data further comprising: synthetic data, pertubated data or a combination thereof.
  • said gradient estimation algorithm comprises one or more types of back-propagation algorithms.
  • the loss function is configured to optimize the explainable neural network by minimizing an error between a prediction of the explainable neural network and a labelled training output of the training data.
  • the loss function comprises a binary and/or categorical cross entropy.
  • the loss function is customizable in relation to the training data.
  • the loss function is adapted to minimize overlap in plurality of partitions associated with the explainable neural network.
  • transforming the explainable neural network using one or more transformations to reconfigure the explainable neural network for adapting to one or more of desired speed, performance, trainability, or criteria that correspond to the explainable neural network is configured to generate both local and global explanations in relation to said predictions; and wherein the global explanations may be generated without said inputs.
  • the global explanation is used to detect areas of strength, weakness and bias in relation to said inputs.
  • a single iteration traversing the explainable neural network provides said predictions and explanations in an ante-hoc manner, absent further simulation, iteration and/or perturbation.
  • the global explanation is enabled by encoding a set of rules in a manner suitable for being interpreted by a machine or read by a user.
  • the set of rules encodes the weights and a structure of the explainable neural network, wherein the set of rules comprises a logic induction mechanism with at least one logic equivalent associated with the structure, wherein said at least one logic equivalent comprises a logic form, a logic clause, a logic statement, a logic assertion, a logic type, a logic expression, and a logic system.
  • said logic induction mechanism is configured to extract casual relationships between an abstraction and a model at any layer or network within the explainable neural network.
  • the set of rules are integrated with information in relation to taxonomy of said inputs, bias detectors, and external knowledge bases.
  • the explainable neural network is configured to be operable at least in part on a low-powered system or chip.
  • the explainable neural network is configured to be operable at least in part on a hardware circuit, the hardware circuit comprising: a flexible architecture, a static architecture, an analogue/digital electronic, a neuromorphic architecture suited for the hardware implementation of connectionist models, or a suitable quantum computing hardware for improved performance.
  • a structure of the explainable neural network is preserved upon embedded in a different system, network, or device, wherein the structure comprises a single module or layer within said different system, networks, or device.
  • the explainable neural network comprise a hybrid neural network and a transducer implementation, or a logically equivalent network structure for combing at least parts of the explainable neural network without loss of functionality.
  • the explainable neural network is configured for integrating one or more convolutional layers for processing images, 3D data, and signals as said inputs.
  • the explainable neural network is adapted to process textual data as said inputs of one or more computable formats and configured to handle syntactic and semantic information from the processed data.
  • the explainable neural network is adapted to process temporal data or sequential data as said inputs, wherein the explainable neural network is configured to forecast trend based on the processed data, wherein the forecasted trends are presentative of recurrences and adapted for explaining one or more cyclical patterns underlying said inputs.
  • the explainable neural network that is used to process temporal data and sequential data is a predictive and recurrent explainable neural network comprising at least one of: a long short-term memory network, a recurrent neural network, a transformer, a Bayesian network, and a Markov model.
  • the explainable neural network that is integrated with at least one convolutional layer is further combined with the explainable neural network that is adapted to process temporal data or sequential data to form a video explainable neural network for processing sequence of images as said inputs.
  • the explainable network is configured to use in combination with any one or a combination of: a generative adversarial network, explainable auto-encoder, explainable auto-decoder, a reinforcement learning model, a causal explainable network, a dialogue and conversational explainable network, and a speech explainable network.
  • said predictions and explanations provided by the explainable neural network, or the combination thereof act as one or more triggers for an external system or device.
  • the explainable neural network is compatible with alternative explainable technologies, wherein the alternative explainable technologies comprise any one or a combination of: model, network, transformers, applications, algorithm, methods or systems.
  • said inputs comprise linear and non-linear data sets, wherein the nonlinear data set is processed using one or more transformations associated with the explainable neural network.
  • the explainable neural network is adapted to process rotational and scale invariance in a combination with non-linear functions for generalizing over said plurality of partitions in at least two dimensions.
  • the explainable neural network is adapted to apply one or more applying one or more transforms.
  • the explainable neural network is adapted to apply at least one of: a Fourier transform, integer transform, real number transform, complex number transform, quaternion transform, octonion transform, Walsh function, state-space transform, phase-space transform, Haar and non-Haar wavelets, generalized L2 function, fractal-based transform, Hadamard transform, fuzzy logic, knowledge graph networks, categorical encoding, difference analysis, normalization, standardization, multi-dimensional Bezier curves, recurrence relations, and causal operators.
  • the transformation function may be a sub-neural network and/or a gradient-descent-based method such as back-propagation.
  • the explanations provided by the explainable neural network does not contribute to additional computing cost provided that any traversable path of the network is identified or predefined.
  • the explainable neural network is configured as a classifier, adapted to perform regression tasks, or simultaneously for classifying and performing recession tasks.
  • said predictions and explanations of the explainable neural network is adapted to display said predictions and explanations based on one or more display options, wherein the said one or more display is configured to receive input from a user or a user operable machinery.
  • the predictions and explanations are transformed using one or more suitable data structure comprising: a hierarchical tree or network, a causal diagram, a directed or undirected graph, a multimedia structure, a set of hyperlinked graph.
  • system for implementing an explainable neural network comprising: an input interface configured for receiving inputs to the explainable network; the explainable neural network configured according to any of aspects or options either directly or indirectly connected to the interface for receiving said inputs, wherein the explainable neural network is trained in accordance; and a display interface configured for displaying predictions and explanations associated with said inputs.
  • the display interface is adapted for displaying classifications associated with the predictions and explanations.
  • the system in response to a null or without input, is configured to provide at least one explanation as output to be displayed.
  • a system for an artificial neural network that is interpretable and explainable, comprising: an input layer which receives an input and identifies one or more input features; a conditional network, comprising: a conditional layer configured to model the input features based on one or more partitions, wherein each of the one or more partitions comprises a rule; an aggregation layer configured to aggregate one or more rules into one or more of the partitions; and a switch output layer configured to selectively pool the aggregated partitions from the aggregation layer with the remaining partitions from the conditional layer; a prediction network, comprising: a feature generation and transformation network comprising one or more transformation neurons configured to apply one or more transformations to the input features; a fit layer configured to combine features which have been transformed by the feature generation and
  • system is further configured to apply an additional transformation within at least one of: the prediction network before the fit layer, at the output, and within the conditional network before the conditional layer.
  • each of the one or more partitions forms at least one local model, and a combination of local models forms a global model, wherein one or more activation paths throughout the partitions are identifiable.
  • the system further comprising a ranking layer configured to identify a score for each of the one or more activation paths wherein the activation path(s) comprise one of a hierarchical and a flat structure.
  • each partition is explainable and interpre table, such that each local model formed from the partitions is explainable and interpretable, and the global model formed from the combination of local models is explainable and interpretable, and wherein an explanation is formed with the output in a single feed forward step.
  • the feature generation and transformation layer is configured to apply at least one of a linear transformation and a non-linear transformation, wherein the transformation functions comprise one or more of polynomial expansions, rotations, dimensional scaling, dimensionless scaling, Fourier transforms, integer/real/complex/quaternion/octonion transforms, Walsh functions, state-space transforms, phase-space transforms, Haar and non-Haar wavelets, generalized L2 functions, fractal-based transforms, Hadamard transforms, Type 1 fuzzy logic, Type 2 fuzzy logic, knowledge graph networks, categorical encoding, difference analysis, normalization, standardization, scaling, multi-dimensional Bezier curves, recurrence relations, causal operators, gradient descent based transformations, and subsets of an explainable neural network.
  • the transformation functions comprise one or more of polynomial expansions, rotations, dimensional scaling, dimensionless scaling, Fourier transforms, integer/real/complex/quaternion/octonion transforms, Walsh functions, state-space
  • the input is received on the conditional network and prediction network simultaneously.
  • the partitions are hierarchically structured.
  • the transformation layer is further configured to perform a plurality of transformations in a transformation pipeline.
  • the transformation pipeline is further configured to perform transformations that analyze one or more temporally ordered data sequences according to the value of one or more variables.
  • system further comprising at least one of a selection, ranking, split, and merge layer implemented at least one of before, in, and after the conditional network.
  • the partitions are non-overlapping.
  • the prediction network fits one or more local models, wherein each local model is linked to a specific partition.
  • system further comprising a loss function applied to minimize or completely eliminate the overlap between partitions and to minimize the error between a prediction and a labeled output.
  • each partition is a local model and a combination of more than one partitions is a global model.
  • the value output layer is configured to present at least one of a predicted value and a classification label based on the input features, and wherein the value output layer further includes an activation function.
  • system is further configured to form the partitions, based on the input, by comparing features from the input to a set of localization values and conditions, wherein the localization values and conditions are identified using an internal and/or external process.
  • the internal and/or external process is a gradient descent method.
  • human knowledge is embedded into the network.
  • the system is further configured to apply one or more gradient descent methods to implement human knowledge and machine-generated knowledge into the partitions.
  • the input data is an output from another neural network.
  • the output is machine readable and is read by a subsequent neural network.
  • the rules are in one of the following formats: IF-THEN, disjunctive normal form, conjunctive normal form, Boolean logic, first order logic, second order logic, propositional logic, predicate logic, modal logic, probabilistic logic, many- valued logic, fuzzy logic, intuitionistic logic, non-monotonic logic, non-reflexive logic, quantum logic, paraconsistent logic or other suitable type of logical system for the expression of logical or similar statements.
  • one or more of the partitions overlap with one another, and wherein the system is configured to identify at least one of a probability or ranking score for each partition, partitions to merge, partitions to split.
  • one or more of the partitions overlap with one another, and wherein the system further comprises a ranking function configured to rank the partitions and select the highest ranking partition when more than one overlapping partition is identified.
  • the coefficients and/or partitions are formed based on at least one of human input, taxonomy information, and ontology information.
  • [0231] in another aspect is a computer implemented method for providing an explainable neural network, comprising executing on a processor the steps of: inputting a set of data into an input layer; partitioning the input based on one or more input features identified in the input, creating one or more partitions; aggregating one or more of the partitions; applying one or more transformation functions to the partitioned input features, providing a set of transformed features; combining the transformed features and identifying one or more coefficients corresponding to the transformed features; compiling one or more equations based on the coefficients and the transformed features; compiling one or more rules based on the equation and the partitions; applying the equations and rules to the set of input features to obtain an output value; and outputting the output value and outputting an explanation of the output value, wherein the explanation accounts for at least one of the coefficients.
  • the method further comprising convoluting the set of data prior to inputting the set of data into the input layer.
  • the method further comprising customizing one or more of the partitions based on human input.
  • the transformation transforms the prediction output using to be structured as one of: (i) hierarchical tree or network, (ii) causal diagrams, (iii) directed and undirected graphs, (iv) multimedia structures, and (v) sets of hyperlinked graphs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

Selon l'invention, une architecture pour un réseau neuronal explicable peut mettre en œuvre un certain nombre de couches pour produire une sortie. La couche d'entrée peut être traitée à la fois par un réseau conditionnel et par un réseau de prédiction. Le réseau conditionnel peut comprendre une couche conditionnelle, une couche d'agrégation et une couche de sortie de commutation. Le réseau de prédiction peut comprendre une couche de production et de transformation de caractéristiques, une couche d'ajustement et une couche de sortie de valeur. Les résultats de la couche de sortie de commutation et de la couche de sortie de valeur peuvent être combinés pour produire la couche de sortie finale. Un certain nombre de fonctions d'activation différentes possibles peut être appliqué à la couche de sortie finale selon l'application. Le réseau neuronal explicable peut être mis en œuvre en utilisant à la fois du matériel informatique à usage général et également de la circuiterie spécifique à l'application contenant des mises en œuvre exclusivement matérielles optimisées. L'invention concerne divers modes de réalisation de XNN qui étendent la fonctionnalité à différents domaines d'application et différentes industries.
PCT/EP2020/082449 2019-11-18 2020-11-17 Architecture pour réseau neuronal explicable WO2021099338A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP20811254.0A EP4062330A1 (fr) 2019-11-18 2020-11-17 Architecture pour réseau neuronal explicable

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201962936716P 2019-11-18 2019-11-18
US62/936,716 2019-11-18
US202062964850P 2020-01-23 2020-01-23
US62/964,850 2020-01-23
US202063021230P 2020-05-07 2020-05-07
US63/021,230 2020-05-07

Publications (1)

Publication Number Publication Date
WO2021099338A1 true WO2021099338A1 (fr) 2021-05-27

Family

ID=73543229

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2020/082449 WO2021099338A1 (fr) 2019-11-18 2020-11-17 Architecture pour réseau neuronal explicable

Country Status (2)

Country Link
EP (1) EP4062330A1 (fr)
WO (1) WO2021099338A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113269310A (zh) * 2021-06-03 2021-08-17 北京邮电大学 基于反事实的图神经网络可解释性方法
CN113609272A (zh) * 2021-08-12 2021-11-05 北京师范大学珠海校区 一种适用于数值推理任务的胶囊网络结构
CN113724110A (zh) * 2021-08-27 2021-11-30 中国海洋大学 一种可解释的深度知识追踪方法、系统及其应用
CN114004340A (zh) * 2021-11-15 2022-02-01 南开大学 一种基于序列变换的循环神经网络可解释性方法、系统
CN114740159A (zh) * 2022-04-14 2022-07-12 成都秦川物联网科技股份有限公司 一种天然气能量计量组分获取方法和物联网系统
CN115131618A (zh) * 2022-07-28 2022-09-30 西安电子科技大学 基于因果推理的半监督图像分类方法
DE102021211503B3 (de) 2021-10-12 2023-02-09 Continental Automotive Technologies GmbH Verfahren zum Überwachen logischer Konsistenz in einem Machine-Learning-Modell und zugehörige Überwachungseinrichtung
GB2609904A (en) * 2021-08-05 2023-02-22 Impulse Innovations Ltd Systems and methods for generating a structural model architecture
EP4198837A1 (fr) * 2021-12-17 2023-06-21 Sap Se Procédé et système d'explication globale de réseaux neuronaux
CN116306773A (zh) * 2022-09-13 2023-06-23 西湖大学 一种可解释的神经网络降维分析系统
US11979697B2 (en) 2021-07-26 2024-05-07 Chengdu Qinchuan Iot Technology Co., Ltd. Methods and internet of things systems for obtaining natural gas energy metering component

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
CHAOFAN CHEN ET AL: "An Interpretable Model with Globally Consistent Explanations for Credit Risk", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 30 November 2018 (2018-11-30), XP081042310 *
JOEL VAUGHAN ET AL: "Explainable Neural Networks based on Additive Index Models", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 5 June 2018 (2018-06-05), XP080887512 *
ZEBIN YANG ET AL: "Enhancing Explainability of Neural Networks through Architecture Constraints", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 12 January 2019 (2019-01-12), XP081493360 *

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113269310A (zh) * 2021-06-03 2021-08-17 北京邮电大学 基于反事实的图神经网络可解释性方法
US11979697B2 (en) 2021-07-26 2024-05-07 Chengdu Qinchuan Iot Technology Co., Ltd. Methods and internet of things systems for obtaining natural gas energy metering component
GB2609904A (en) * 2021-08-05 2023-02-22 Impulse Innovations Ltd Systems and methods for generating a structural model architecture
CN113609272A (zh) * 2021-08-12 2021-11-05 北京师范大学珠海校区 一种适用于数值推理任务的胶囊网络结构
CN113724110A (zh) * 2021-08-27 2021-11-30 中国海洋大学 一种可解释的深度知识追踪方法、系统及其应用
DE102021211503B3 (de) 2021-10-12 2023-02-09 Continental Automotive Technologies GmbH Verfahren zum Überwachen logischer Konsistenz in einem Machine-Learning-Modell und zugehörige Überwachungseinrichtung
CN114004340A (zh) * 2021-11-15 2022-02-01 南开大学 一种基于序列变换的循环神经网络可解释性方法、系统
CN114004340B (zh) * 2021-11-15 2024-06-07 南开大学 一种基于序列变换的循环神经网络可解释性方法、系统
EP4198837A1 (fr) * 2021-12-17 2023-06-21 Sap Se Procédé et système d'explication globale de réseaux neuronaux
CN114740159B (zh) * 2022-04-14 2023-09-19 成都秦川物联网科技股份有限公司 一种天然气能量计量组分获取方法和物联网系统
CN114740159A (zh) * 2022-04-14 2022-07-12 成都秦川物联网科技股份有限公司 一种天然气能量计量组分获取方法和物联网系统
CN115131618A (zh) * 2022-07-28 2022-09-30 西安电子科技大学 基于因果推理的半监督图像分类方法
CN115131618B (zh) * 2022-07-28 2024-09-24 西安电子科技大学 基于因果推理的半监督图像分类方法
CN116306773A (zh) * 2022-09-13 2023-06-23 西湖大学 一种可解释的神经网络降维分析系统
CN116306773B (zh) * 2022-09-13 2024-03-26 西湖大学 一种可解释的神经网络高维数据降维分析系统

Also Published As

Publication number Publication date
EP4062330A1 (fr) 2022-09-28

Similar Documents

Publication Publication Date Title
US11055616B2 (en) Architecture for an explainable neural network
AU2021399965B2 (en) Explainable transducer transformers
WO2021099338A1 (fr) Architecture pour réseau neuronal explicable
LeCun A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27
Yu et al. A survey on neural-symbolic learning systems
Lieto et al. Conceptual spaces for cognitive architectures: A lingua franca for different levels of representation
US20200104726A1 (en) Machine learning data representations, architectures, and systems that intrinsically encode and represent benefit, harm, and emotion to optimize learning
Van-Horenbeke et al. Activity, plan, and goal recognition: A review
US11295199B2 (en) XAI and XNN conversion
Riley et al. Integrating non-monotonic logical reasoning and inductive learning with deep learning for explainable visual question answering
Olier et al. Re-framing the characteristics of concepts and their relation to learning and cognition in artificial agents
Russo et al. Knowledge acquisition and design using semantics and perception: a case study for autonomous robots
Talukdar et al. Artificial intelligence in healthcare industry
TWI803852B (zh) 可解釋人工智慧及可解釋神經網路轉換
Vanani et al. Deep learning for opinion mining
Scharei et al. Knowledge representations in technical systems--a taxonomy
Gibaut et al. Neurosymbolic AI and its Taxonomy: a survey
Jeyachitra et al. Machine learning and deep learning: Classification and regression problems, recurrent neural networks, convolutional neural networks
TWI810549B (zh) 可解釋的神經網路、相關的電腦實施方法、及用於實施可解釋的神經網路之系統
Yousif et al. Critical Review of Neural Network Generations and Models Design
Koh et al. Artificial Intelligence (AI) Fundamentals for the Display Industry: A Review
Zhong et al. Disentanglement in conceptual space during sensorimotor interaction
WO2021115929A1 (fr) Conversion de xai et de xnn
Bhattacharyya et al. A knowledge-driven layered inverse reinforcement learning approach for recognizing human intents
Orten Machine Ethics for Autonomous systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20811254

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020811254

Country of ref document: EP

Effective date: 20220620