FR3118244A1 - Method and device for diagnosing anomalies - Google Patents

Abstract

The invention proposes an anomaly diagnosis device (100) comprising: - a determination of the class to which an input data corresponding to a query point belongs, by applying a model (10) using decision trees, - a decomposition (14) of the leaves of the decision trees into pure boxes, each corresponding to a region of the space of input characteristics of the input data; - a calculation of a counterfactual point (17) representing the virtual point closest to the query point belonging to a target class, from the decomposition; - a calculation of a change vector (18) representing the minimum changes to be applied to the input data so that the input data is classified in the target class by the classification model, from the difference between the point counterfactual Y and the input query X, the change vector dX. Figure for the abstract: Fig.1

Description

Method and device for diagnosing anomalies

The invention generally relates to the field of classification and in particular to a device and a method for diagnosing anomalies using one or more decision trees.

It is common in existing anomaly detection devices to use machine learning based on the use of “set of decision tree” type learning models.

Such devices are conventionally used in industrial systems, such as systems for testing and diagnosing defective objects or equipment on production lines, systems for diagnosing fraud in the banking or tax field, or diagnostic systems medical or aid to medical diagnosis. In an example application, from fault diagnostic devices to industrial systems, the manufacturing processes are complex and involve many steps such that there is a high risk that a fault (defect) will be introduced before completion. of the manufacturing process.

In such applications, the anomaly diagnosis device based on set-tree type machine learning models must be able to intelligibly explain the decision-making of the anomaly detection system. For example, in medical diagnostic devices, “black box” type learning models are not homologable inside medical devices because it is not possible to link the decision making to the parameters of the model.

Thus, the ability to explain the model's decision-making relies on the interpretability of the outputs (decisions) delivered by the "set of decision tree" type models.

A definition of interpretability based on the notion of counterfactual explanations has been proposed for example in C. Molnar, “Interpretable, Machine Learning, 2019, A Guide for Making Black Box Models Explainable”. This definition is based on the notion of counterfactual explanations, a 'counterfactual explanation' of a prediction delivered by a learning model describing the smallest change in the values of input characteristics of the model which modifies the model prediction at a predefined output value .

To detect anomalies, binary classification models (especially ensemble-tree type models such as XGBoost1 2 or lightGBM3 4) are usually trained to classify data into two classes, for example a ”presence of anomaly” class versus an “absence of anomaly” class. The anomaly diagnosis must thus be able to provide an answer corresponding to the interpretation of the decision of the binary classification model to place the data characterizing the anomaly in the “anomaly” class.

Two categories of solutions have been proposed in the state of the art to provide an interpretation of the decisions of the classification models including:
- methods, known as sensitivity analysis, which statistically quantify the influence of a characteristic or group of characteristics on the output of the model. However, such methods, like variable selection techniques, only give the user information about the characteristics that are important for the model, i.e. the characteristics that influence the decision. , without providing explanatory information with respect to a specific entry. Consequently, such methods do not make it possible to make a point-by-point explanation and are consequently not suitable for diagnosing an anomaly.
- Point-to-point explanation techniques. Given an input to the model, such techniques are able to determine parameters indicating why the model made a decision regarding that input. Point-to-point explanation techniques include so-called 'model agnostic' methods which view the decision model as a black box and make an independent analysis of the internal peculiarities of the model, and so-called 'model-specific' methods. to the model” (”model specific”) which deconstruct the model taking into account its particularities to provide an explanation of the decision taken.

Among the model-agnostic methods, the most commonly used methods are “model surrogate” type approaches which approximate the model by a simpler and more interpretable model either globally (C.Molnar, Interpretable , Machine Learning, 2019, A Guide for Making Black Box Models Explainable, section 5.6), or locally around the prediction to be explained (as described for example in M. T. Ribeiro, S. Singh, and C. Guestrin, "why should i trust you?" explaining the predictions of any classifier," in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135-1144). ability to approximate the model by an interpretable surrogate model In surrogate model-type approaches, the models are simple models (e.g. decision trees, linear models, additive models generalized ifs etc.) which do not allow complex models to be approximated satisfactorily, which makes global substitution models little used. A similar problem arises for local surrogate models. For example, the local approximation is of the kernel smoothing type in M. T. Ribeiro, S. Singh, and C. Guestrin, \" why should i trust you?" explaining the predictions of any classier," in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135-1144. This approximation is very sensitive to the size of the neighborhoods, and therefore very unstable with respect to the choice of the bandwidth parameter of the kernel ('kernel') used for smoothing ('smoothing') In general, it is difficult to find parameterizations of the local substitution models which guarantee a good approximation of the original model. , it requires a parametrization around each point to be explained, which makes the approximation complex, resulting in great instability of the explanations determined according to the chosen parametrizations.

Model specific methods dedicated to sets of decision trees analyze, for a given input, the contribution of each feature to the final decision of the model. They provide a type of explanation that is not very suitable for anomaly diagnosis, such methods, similar to sensitivity analysis type methods, mainly analyzing the characteristics that most influenced the model in its decision. However, the model tends to look at the same characteristics regardless of the class of the input data (“abnormal” or “normal”). The explanation produced therefore does not give a characterization of the anomaly in terms of abnormal values of the input characteristics, i.e. it does not explain what has changed at the level of the measurements/characteristics of input relative to data classified as “normal” by the model. It is therefore not possible with this type of method to obtain a targeted diagnosis of the fault in order to identify its exact cause. The only information the operator obtains is a ranking ('ranking') in descending order of importance of the characteristics that the model used to produce his decision.

Thus, the solutions proposed for detecting anomalies based on models of the set of decision trees type are blind to the decision-making process of the model (in particular “model agnostic” type methods), produce imprecise explanations (in particular global surrogate model approaches) and/or unstable (local surrogate model approaches) due to the lack of knowledge to properly approximate the model. They are generally too imprecise in relation to the definition of interpretability posed. Moreover, they do not make it possible to precisely characterize the changes in the input characteristics which caused the decision-making of the model.

There is therefore a need for an improved anomaly diagnosis method and device capable of accurately characterizing the changes that caused the anomaly.

General definition of invention

The invention improves the situation by proposing an anomaly diagnostic device comprising a classification unit configured to determine, in response to a request received comprising an input datum and a target class, the class to which the datum belongs. entry among a set of classes, by applying the input datum to a multiclass classification model, the input datum corresponding to a query point. The classification model uses a set of decision trees, the input data comprising a set of input features, the space of input features having a given number of dimensions. Each decision tree comprises a set of nodes defining a tree structure from a root node to a set of leaves, each leaf of the decision tree comprising a set of scores associated with each class, each decision tree being applied to the input datum for determining an elementary classification decision from the scores associated with the leaves of the decision tree, the classification unit applying an aggregation function to the elementary classification decisions determined by each of the trees of the model to determine the class of the input data.

The anomaly diagnosis device further comprises:
- a decomposition unit configured to decompose at least part of the set of leaves associated with the decision trees of the model into multidimensional boxes, called pure boxes, from the input request, the pure boxes corresponding to a collection of intervals, each pure box corresponding to a region of the input feature space, associated with a vector score determined from the scores associated with the leaves of the decision trees;
- a counterfactual point calculation unit configured to calculate a counterfactual point (Y) representing the virtual point closest to the query point belonging to the target class, from the decomposition into pure boxes;
- a change vector calculation unit configured to compare the counterfactual point with the input query and determine the difference between the counterfactual point (Y) and the input query (X), which provides a change vector ( dX) representing the minimum changes to be applied to the input data for the input data to be classified in the target class by the classification model, the change vector (dX) having the same size as the characteristics of entrance.

In one embodiment, the decomposition unit can determine the pure box decomposition as a derived tree structure, whose nodes at a depth d contain the regions corresponding to the pure boxes, the depth d corresponding to a dimension of the input feature space.

In one embodiment, the plurality of decision trees of the model may comprise intersecting leaves, the decomposition unit further being configured to determine the decomposition of the model into pure boxes as regions of maximum intersection , called maximum intersection boxes, a maximum intersection box representing a region of the model having a uniform score over the entire region resulting from the aggregation of the scores of the intersecting leaves to form the region, by applying the aggregation function.

The decomposition unit can be configured to determine the maximum intersection boxes of the model by induction on the dimensions of the input feature space, applying a plurality of iterations, each iteration corresponding to one of the dimensions , the decomposition unit providing the decomposition into pure boxes of the model in the form of maximum intersection boxes dimension by dimension.

In one embodiment, the decomposition unit may be configured to determine, from a maximum intersection box decomposition of the restriction of the model to the first d dimensions of the model, a maximum intersection box decomposition of the restriction of the model to the first d+1 dimensions of the model, d designating a given dimension of the model, a restriction of the model to the first d dimensions designating the restriction of the set of boxes corresponding to the leaves of the decision trees of the model to the first d model size.

In one embodiment, the maximum intersection boxes are maximum intersection boxes associated with the restriction of the model to the first d+1 dimensions.

Advantageously, the model can be decomposed into a set of pure boxes comprising at most elements, N denoting the number of leaves of the decision trees of the model and representing the dimension of the input feature space applied to the model.

In one embodiment, the counterfactual point calculation unit can be configured to calculate the counterfactual point, by determining, for each pure box, a point on the surface of the pure box located at a minimum distance between the query point representing the input data and the pure box.

In one embodiment, the point Z of the surface of the pure box Bi located at a minimum distance between the query point X representing the input data and the pure box Bi is given by:

In one embodiment, the diagnostic device can be configured to calculate a lower bound and an upper bound for each node of the derived tree structure corresponding to the decomposition into pure boxes, each level of the derived tree structure corresponding to a dimension of the model, the lower bound representing a lower bound on the distance to the query point X of all the pure boxes lying below the node, the upper bound representing the upper bound on the distance from the counterfactual point to the query point, the device being configured to stop the construction of the derived tree structure at the level of the nodes for which the lower bound is greater than the upper bound.

In one embodiment, for a given node of the derived tree structure, the lower bound is equal to the distance from the query point to the pure box corresponding to this node.

The counterfactual point for a query point X and a target class "j" can be determined from the pure boxes located at depth D of the tree structure calculated for the model according to the equation:

There is further provided an abnormality diagnosis method comprising the steps of:
- in response to a request received comprising an input datum and a target class, determining the class to which the input datum belongs among a set of classes, by applying the input datum to a multiclass classification model, the input data corresponding to a query point,
the classification model using a set of decision trees, the input data comprising a set of input features, the space of input features having a given number of dimensions,
each decision tree comprising a set of nodes defining a tree structure from a root node to a set of leaves, each leaf of the decision tree comprising a set of scores associated with each class, each decision tree being applied to the input datum for determining an elementary classification decision from the scores associated with the leaves of the decision tree, an aggregation function being applied to the elementary classification decisions provided by each of the trees of the model to determine the class of the input data;
- decomposing at least part of the set of leaves associated with the decision trees of the model into multidimensional boxes, called pure boxes, from the input query, the pure boxes corresponding to a collection of intervals, each pure box corresponding to a region of the input feature space, associated with a vector score determined from the scores associated with the leaves of the decision trees;
- calculate a counterfactual point (Y) representing the virtual point closest to the query point belonging to the target class, from the decomposition into pure boxes;
- determine the difference between the counterfactual point (Y) and the input query (X), which provides a change vector (dX) representing the changes to be applied at least to the input data so that the input is classified into the target class by the classification model, with the change vector (dX) having the same size as the input features.

The change vector provided by the anomaly diagnosis device and method makes it possible to interpret the decisions of the classification model at a point in the space of the input characteristics by comparing this point with the closest point belonging to a another class (belonging to a class being defined by the classification model).

The embodiments of the invention provide an exact resolution and a scalability of the algorithmic problem underlying the search for the nearest counterfactual point, within the framework of classification models of the “set” type. of decision trees” potentially comprising a very large number of trees.

Brief Description of Figures

Other characteristics, details and advantages of the invention will become apparent on reading the description given with reference to the appended drawings given by way of example and which represent, respectively:

There represents an anomaly diagnosis device, according to certain embodiments;

There represents an example of a decision tree with the associated decision space, according to certain embodiments;

There illustrates the counterfactual point for an example classification problem.

There illustrates a decomposition of multidimensional boxes into boxes of maximum intersection, according to one embodiment;

There illustrates the decomposition into boxes of maximum intersection of a set consisting of the union of two boxes, according to one embodiment;

There illustrates such a property of intersecting boxes, according to an exemplary embodiment;

There illustrates the steps of the decomposition process into pure boxes, in an exemplary embodiment.

The diagram of the graphically represents an example of a derived tree structure obtained from the decomposition of the model, according to certain embodiments;

There is a flowchart representing the method for diagnosing anomalies, according to the embodiments of the invention; And

There is an example of a high-level representation of the operation of the decomposition process with trace feedback, according to one embodiment.

In addition, the detailed description is augmented by Appendix A. Appendix A includes examples of methods described in pseudo-code (Algorithm 1 to 5), which can be implemented in certain embodiments.

This Appendix is separated for the purpose of clarification, and for ease of cross-referencing. It is an integral part of the description, and can therefore not only be used to better understand the present invention, but also contribute to its definition, where appropriate.

In Appendix A, notation conventions, usual for those skilled in the art, are used, such as the conditional statements "if-then-else" (if-then-else), "while-do" (as long as -do), “for-do” (for-do). Algorithms involve different variables associated with names.

This description is likely to involve elements susceptible to protection by author's rights and/or copyright. The rights holder has no objection to the identical reproduction by anyone of this patent document or its description, as it appears in official documents. For the rest, he fully reserves his rights.

Detailed description of the request

The embodiments of the invention provide an anomaly diagnosis device and method based on an explicit and exact calculation of the decision regions of a “set of trees” type multiclass classification model using a geometric characterization of the decision space of such a model, in response to a request received comprising an input datum (represented by a query point) and a target class. The anomaly diagnosis device and method determines the class to which the input data belongs among a set of classes, by applying the input data to the multiclass classification model.

From the geometric characterization of the decision space, the device and the method according to the embodiments of the invention determine a counterfactual point. representing the closest virtual point, in terms of Euclidean distance in the model input feature space, to the query point belonging to the target class.

Once the virtual point is extracted, a change vector representing the minimum changes to be applied to the input data for the input data to be classified in the target class by the model is determined, which provides the explanation of model decision. Such a vector of change can then be used to diagnose the anomaly.

There shows a fault diagnostic device 100 according to some embodiments.

The anomaly diagnostic device 100 comprises a classification unit 11 configured to determine, in response to a request comprising an input datum and a target class and to a target class , the class to which the input data belongs among a set of classes, using a multiclass classification model 10 of the set of decision trees type.

The input data corresponds to a query point . The input data includes a set of input characteristics. The input feature space has a number of given dimensions.

The anomaly diagnostic device 100 according to the embodiments of the invention makes it possible to determine an exact response to a request of the type: "for a given point belonging to a class" ”, what is the “virtual” point of the class ” ” ( ) nearest in terms of Euclidean distance in the input feature space?

As used here, a "virtual" point refers to a point in the input feature space that does not necessarily exist in the Model 10 training set, but is constructed using the characterization geometry of the model.

The anomaly diagnosis model 10 is a binary classification model of the “decision tree set” type, such as for example XGBoost or lightGBM.

Given multidimensional input numerical characteristics characterizing an object, the classification model 10 makes it possible to classify this object among the classes of the set of classes. For example, if the classification model 10 is a binary classification model, the set of classes may include a class "presence of anomaly" and a class "absence of anomaly".

The classification model 10 of the set of decision trees type is previously trained using training data.

Each decision tree of the classification model includes a set of nodes defining a tree structure from a root node to a set of leaves. Each leaf of the decision tree includes a set of scores associated with each class. The input data is applied to each decision tree of the model, which provides an elementary classification decision from the scores associated with the leaves of the decision tree. The classification unit 11 applies an aggregation function to the elementary classification decisions provided by each of the decision trees of the model to determine the class of the input data.

The diagnostic device 100 includes a decomposition unit 14 to determine the geometric characterization of the model 10.

The decomposition unit 14 is configured to decompose at least part of the set of leaves associated with the decision trees of the model into multidimensional boxes, called “pure boxes”, from the input request. The pure boxes correspond to a collection of intervals, each pure box corresponding to a region of the input feature space, associated with a vector score determined from the scores associated with the leaves of the decision trees. The definition set of model 10 of type "set of trees" is exactly the union of the boxes corresponding to the leaves of the trees.

The diagnostic device 100 comprises a counterfactual point calculation unit 17 configured to calculate a counterfactual point (Y) representing the closest virtual point, in terms of Euclidean distance in the space of the input characteristics, of the query point belonging to the target class, from pure box decomposition.

The diagnostic device 100 includes a change vector determination unit 18 configured to determine the difference between the counterfactual point and the query point , which provides a vector of change representing the minimum changes to be applied to the input data for the input data to be classified in the target class by the classification model. The vector of change having the same size as the input features. The vector is representative of the underlying causes of the classification decision made by the model 10 for the input datum and can be used to interpret the model classification decision.

The anomaly diagnosis device 100 according to the embodiments of the invention thus uses a geometric representation of a classification model of the “set of decision trees” type 11 obtained from the decomposition into pure boxes of the model .

The anomaly diagnostic device 100 according to the embodiments of the invention advantageously provides a rapid and exact response to such an input request represented by the characteristics . It is particularly suited to the use of arbitrarily large classification models.

• désigne la dimension de l’espace des caractéristiques d’entrée appliquées au modèle 10 ;
• Les caractéristiques d’entrée présentées en entrée du modèle sont représentées par un vecteur ∈ ;
• désignele nombre de classes prédéfinies dans l’ensemble d’apprentissage ;
• désigne le modèle de classification 10. Un modèle définit une correspondance (mapping) entre l’espace des caractéristiques d’entrée et une classe:

.To facilitate understanding of the following description of certain embodiments and definitions used in connection with such modes, the following notations are defined in connection with Classification Model 10:

• denotes the dimension of the input feature space applied to the model 10;
• The input characteristics presented as input to the model are represented by a vector ∈ ;
• denotes the number of predefined classes in the training set;
• denotes classification model 10. A model defines a mapping between the input feature space and a class:

.

Furthermore, a decision tree designates a binary tree comprising a root node from which starts a tree structure comprising a set of nodes. A decision tree node (including the root node) contains an input feature index and a threshold value. When a piece of data arrives at a given node of the decision tree, it goes to the left branch of the node if the value of the characteristic at the index associated with the node is lower than the threshold, and to the right branch of the node if the value of the characteristic at the index associated with the node is greater than the threshold associated with the node. This process of traversing the tree by the datum considered is repeated until it arrives in a leaf of the tree (being part of the root node). A leaf of the decision tree includes a score and a class. The score and the class associated with the leaf of the tree in which a datum arrives are then recorded and associated with the datum.

It should be noted that a characteristic index is not constrained to appear only once and only once in a path between the root node and a leaf of the tree. An arbitrary number of nodes inside the path can thus correspond to this index (including a null number).

Given a depth tree , a leaf of the tree is characterized by a succession of at most tests at each of the nodes located on the path between the root node and the considered leaf.

It is possible to set an arbitrary tree depth regardless of the number of dimensions . When (the depth of the tree is strictly greater than the number of dimensions), for a given leaf, a plurality of tests can be performed along the same dimension on the path between the root node and this leaf.

To test membership in a leaf, a set of interval tests of the type can be performed:
" ∈ [ a, b ]? » with a ∈ RU {−∞} and b ∈ RU {−∞} according to each dimension .

A leaf of a tree therefore geometrically represents a multidimensional “box” each side of which is perpendicular to a coordinate axis. Some faces do not exist if the test interval associated with a coordinate is open on one or both sides. In the remainder of the description, a leaf F _i of a tree will be denoted as a collection of D intervals, according to equation 1:

Furthermore, a decision tree will be denoted hereinafter by f and the -th leaf of the tree will be designated by .

Each leaf of a tree is associated with a couple (score, class) noted ( S [ , Class[ ]), with class [ ] ∈ 1 , ... , K, meaning that the leaf vote for the class [ ] with an S- score [ for this class, and 0 for the other classes.

There shows an example of a decision tree and the associated decision space. By construction, the “boxes” corresponding to the leaves of the same tree do not intersect with each other. Intersections occur when “set of trees” type models are considered. The example tree shown only generates open boxes. The sheet “F1”, for example, is opened on the left following “d1” and at the bottom following “d2”. The sheet “F6” is open on the right following “d1”, and closed on the three other sides.

To obtain a box closed on all sides, at least two tests by coordinates which induce neither contradiction (for example, ”x1 > 0.5 and x1 < 0.3”), nor useless verification (for example, ”x1 > 0.3 and x1 > 0.5”) are to be made. The leaf ”F5”, for example, has an unnecessary test, ”x2 < 0.8”, because it is tested ”x2 < 0.4” at the next node on the ”path” from the root node to the leaf ”F5”. More generally, for a given dimension, a path of the tree has at most two “useful” tests (ie, tests inducing neither contradiction nor useless verification), defining a closed interval for such a dimension.

In the rest of the description, a simplified notation will be used to designate the score of each sheet as a vector , such as :

And

An input data, corresponding to a query point , applied to the classification model 10 follows a single path in a tree until arriving in a single sheet of the tree including a score . To designate the score associated with the leaf of the tree which is reached for an input request , the following notation is used:

Using leaf membership indicator functions, the function can be reformulated according to equation (2):

As used here, a ‘decision tree collection’ refers to a set of models such as XGBoost models, LightGBM models, random forest models, and decision trees.

A classification model (10) type “collection of decision trees” is defined as a set of decision trees , equipped with a vector aggregation function of shape:

The vector aggregation function is used to aggregate the individual decisions (elementary classification decision) from each of the trees in the model into a single decision.

Given an input characteristic , the exit provided by a model in response to an input request , representing the decision made by the model on , is calculated by:

For example, for classification models of type ”XGBoost”, the aggregation function can be of the form:

The output of the model is therefore a vector of size comprising a set of components, each component being associated with a class among the classes each class , the vector component associated with the k-th class comprising class membership information indicating whether the input data belongs or not to the class (“belief” of the model that the data belongs to the class ). The class membership information may for example be a probability. In one embodiment, the model F determines that a datum is classified in a class (ie the class membership information indicates that the input data belongs to the class ) if and only if:

The decomposition unit 14 is configured to determine a geometric reformulation of a model as a collection of sheets/boxes { } , each associated with a vector score . In the rest of the description, the notation given by the following formula (3) will be used:

It is considered a classification model (10), a request associated with an input data , a class and a target class , the query consisting of "determine the virtual point of the class closest in Euclidean distance to a given element of style ". The virtual point representing the "counterfactual point of the class associated with a point is defined by the following formula (4):

To determine the response to the query, the anomaly diagnostic device 100 uses the decomposition unit 14 to determine the decision regions of the model in the space of the input characteristics of the input datum . The decision regions of the model correspond to the regions within which the model makes one and only decision regarding the classification of the elements of the region. Such regions are also called “pure regions” of the model.

Fig. 2B illustrates the notion of counterfactual point for a binary classification problem between a first class C1 versus a second class C2. There represents the decision regions of the model for the two classes C1 and C2. For three points of the class C1 denoted (P1; P2; P3), the points closest to the target class C2 in the decision space of the classification model are sought. These points are denoted respectively (CF1; CF2; CF3) on the diagram of the and correspond to the counterfactual points associated respectively with the points (P1; P2; P3).

To facilitate the understanding of the processing implemented by the decomposition unit 14, the following definitions and properties are provided in relation to the notion of “pure region”.

is a pure region associated with a model F if:

Also, if Class( )=k, the region West called “pure region of class ka associated with the model F ” . In the rest of the description, the notation Class( )=ks will be used to indicate that the pure region W is a pure region of class k .

The characterization of the decision space of a model F denotes a decomposition of the definition space of F into distinct pure regions, as implemented by the decomposition unit 14.

Given a model of definition set E _F R ^D , a decomposition into pure regions of , as implemented by decomposition unit 14, denotes a decomposition such as:

- is a pure region associated with ;

- ;

- .

It should be noted that such a decomposition is not unique.

The distance from a point to a region is defined by:

(5)

Point denotes then the point of the region closest in Euclidean distance to .

Given a model and an associated pure region decomposition , in some embodiments, the counterfactual point calculation unit 17 may determine the counterfactual point of the class associated with a point , as defined by equation (4), by solving the following equation (6):

Advantageously, the decomposition unit 14 can be configured to determine a pure region decomposition of the model definition set F, such that the distance to a pure region, as defined by equation (5) , is exactly and efficiently computable, which makes it possible to determine a geometric reformulation of a classification model F (10).

For a model of the “collection of decision trees” type, the decomposition unit 14 can further be configured to determine a decomposition into pure regions in the form of multidimensional “boxes”, as defined by equation (1) of Annex (A). Such a decomposition is also called “pure box decomposition”. It should be noted that the definition set of a model F of type “set of trees” is the union of the boxes corresponding to the leaves of the trees.

Given a model F of the “set of trees” type, the decomposition unit 14 can thus determine a decomposition into pure boxes of the set of leaves of the classification model F.

It should further be noted that due to the aggregation function g associated with a model, and the potential intersection between leaves belonging to different trees, the boxes corresponding to the leaves are not necessarily pure regions associated with the model.

Thus in one embodiment, the decomposition unit 14 can be configured to also perform a decomposition into maximum intersection boxes associated with the model F considered.

As used here, a region is said to be the “maximum intersection” region associated with a “set of trees” F model if:

- there is a subset of leaves ⊂ such as ,

- for any sheet ,

A region of maximum intersection associated with a model of type “set of trees” designates a pure region associated with this model. Indeed, by construction, a maximal intersection region has a uniform score over the entire region resulting from the aggregation of the scores of the intersecting leaves to form this region:

As used herein, a “maximum intersection box” associated with a “set of trees” model F denotes a region of maximum intersection of F of box type.

Given a model of the “shaft assembly” type comprising leaves , as used here, a maximum intersection box decomposition of denotes a decomposition such as:

- ∀ n ∈ {1, . . . , NOT}, is a maximum intersection box associated with ;

- ;

- ,

There illustrates a decomposition of multidimensional boxes into a maximum intersection box, according to an exemplary embodiment. On the , it is considered a model with three sheets represented by the boxes B1, B2, B3 on the left diagram of the . The right diagram of the shows a decomposition of the three sheets into 11 regions of maximal intersection of the “box” type.

There illustrates the decomposition into maximum intersection boxes of a set consisting of the union of two boxes, B1 and B2 performed by the decomposition unit according to certain embodiments. A first phase of decomposition of the boxes of the definition set B1 and B2 into boxes of maximum intersection is carried out, which gives three regions of maximum intersection R1, R2 and R3. A second phase of decomposing the maximal intersection regions R1, R2 and R3 into maximal intersection boxes W1, W2, W3, W4 and W5 (R1 is decomposed into W1 and W2; R2 is decomposed into W3 and R3 is decomposed into W4 and W5). It should be noted that the regions of maximum intersection are not necessarily boxes but can be disjoint assemblies of boxes of maximum intersection such as the region R1 which breaks down into two boxes of maximum intersection W1, and W2.

As shown in the diagram of the , the regions Ri are not necessarily boxes, but can also be disjoint assemblies of boxes, such as the regions R1 and R3 which break down into two boxes of maximum intersection each (W1 and W2 for R1; W4 and W5 for R3 ).

More generally, an intersection between two D-dimensional boxes has a decomposition comprising 2D+1 or less than 2D+1 boxes.

Moreover, given a model F of type “set of trees” comprising N leaves and whose definition set is included in R ^D , it is possible to find a decomposition into pure boxes of the leaves of F counting at most (2N -1) ^D elements.

In one embodiment, the decomposition unit 14 can be configured to determine the maximum intersection boxes of F (which are also pure regions associated with F) by recurrence, dimension by dimension.

In one embodiment, the decomposition unit 14 can be configured to determine, from a decomposition into maximum intersection boxes, the restriction of the model F (the model F being a model of the “set of trees” type ”) in the first ”d” dimensions, a maximal intersection box decomposition of the restriction of F to the first ”d+1” dimensions (heredity property).

As used here, a restriction of a "set of trees" F-model to first dimensions designates the restriction of the set of boxes corresponding to the leaves of F to the first d dimensions. The restriction of a model F to the first d dimensions will be noted below .

It is considered a model F of type “set of trees” whose definition set is included in and comprising N sheets .

To demonstrate the inheritance property that can be exploited by decomposition unit 14, we first consider the case of initialization D=1.

When the dimension D is equal to one (D=1), the decomposition unit 14, to implement a decomposition into maximum intersection boxes, can determine the maximum intersections resulting from the intersection of N intervals. Although the problem of "determining the maximum intersections resulting from the intersection of N intervals" may seem combinatorial (given m intervals among N, it must be checked whether these intervals have a common intersection area, which would amount to analyze the existence of potential intersections), in practice, the combinatorial aspect can be circumvented by considering that a maximum intersection zone begins or ends at a point if and only if at least one interval among N begins or ends at this point. For N intervals, there are at most N beginnings and N endings of intervals, which consequently creates at most 2N − 1 regions of maximum intersection (the last interval to end does not create a new zone of maximum intersection but ends the last zone, hence the “- 1”).

As a variant, to find the maximum intersection zones, in the case D=1, the decomposition unit 14 can apply a method consisting in ordering the 2 N values corresponding to the beginning and the end of the N intervals, and in creating a new maximum intersection area each time an interval begins or ends. A boolean matrix A of size N (2N-1) can thus be created, such that its component (i,j) denoted A(i,j) can have a true value (i.e. A(i,j)==TRUE, “true” meaning “true” ) indicating that the i -th interval is present at the j -th region of maximum intersection or a "false" value otherwise (ie A(i,j)==FALSE, "false" meaning "false"). Matrix A can be very large. In general, the matrix A has a very sparse structure (“sparse” in Anglo-Saxon language) which allows its explicit construction whatever the number N of leaves of the model.

When the dimension D is strictly greater than one (D>1), a maximum intersection box decomposition is considered for a dimension d < D defined by:

From such a decomposition, the decomposition unit 14 can determine a maximum intersection box decomposition at the dimension .

A box of any size indexed by will be noted below:

, ..., }

Moreover, the n-th sheet of the model will be noted below:

, ..., }

The method for determining the maximum intersection zones can be implemented for example according to the Intersect1D procedure described in the form of pseudo-code in Algorithm 1 of appendix B.1.

Such a method for determining the maximum intersection zones is implemented to calculate the maximum intersection regions associated with N 1D intervals. The nth constructed maximal intersection region is defined by the interval and is formed by the intersection of the intervals of the set whose index is contained in .

The method for determining the maximum intersection zones uses as input a collection of N intervals .

The method for determining the maximum intersection zones also uses:

- The set P of events corresponding to the beginning or the end of an interval, with ;

- The set R containing the indices of the intervals associated with the events of P, with ;

– The index and the whole sorted resulting from sorting in ascending order of the set P: ;

– A Boolean vector Q such that Q(n)==TRUEif and only if the event I(n) corresponds to the beginning of an interval. If so, it is an interval R[I][n];

– A Boolean vector, noted “Started” (“started” being an Anglo-Saxon term meaning “started”) of size N whose values are initialized to FALSE such that Started[n]==TRUE if and only if the event happened and the event has not happened yet.

- a variable id.n initialized to 1 ( id.n 1);

- a variable ind.unique initialized to 1 ( ind.unique 1 );

- a boolean matrix A of size N (2N-1) initialized to FALSE.

The maximum intersection area determination process determines and outputs the maximum intersection areas:

- with And

- with .

Given a box = , the box is called restriction of to first dimensions ( d ≤ D ). By analogy, the leaf restriction to first dimensions is noted .

In the remainder of the description, considering the whole, denoted , including the indices of the sheets of the model restricted to first dimensions that have a nonzero intersection with the box , the following property can be considered to add a dimension:

If two boxes And do not intersect along a dimension d, with , that is, if , SO :

.

Such a geometric property follows directly from the fact that a box is, for one dimension datum, located between the two normal vector hyperplanes passing respectively through the points And , Or denotes the unit vector associated with the -th coordinate axis.

There illustrates such a property. On the left diagram of the , there shows two boxes B1 and B2 which intersect along dimension 1 but do not intersect along dimension 2. On the contrary, on the right diagram of the , boxes B1 and B2 intersect in dimension 2 but do not intersect in dimension 1. In both cases, the intersection of B1 and B2 is zero.

Thus, two boxes that do not intersect along one dimension have zero intersection (even if they intersect along the other dimensions).

In some embodiments, the decomposition unit 14 may determine the intersection boxes , independently, from each of the maximum intersection boxes . For a maximum intersection box , the method of appendix A1.1 (Algorithm 1 in pseudo-code) can for example be applied to the intervals , to extract a maximum intersection box decomposition from the set of leaves restricted to d+1 dimensions.

The maximum intersection interval decomposition of the boxes of the set restricted to dimension d, i.e. intervals , will be noted below:

By applying the process of appendix A1.1, described in pseudo-code by Algorithm 1, it comes:

, .

In the remainder of the description, the following notation will be used to designate the maximum intersection boxes :

The maximum intersection boxes are maximum intersection boxes associated with the restriction of to first dimensions .

The decomposition , determined by decomposition unit 14, from all boxes of maximum intersection is a maximum intersection box decomposition of the leaves restricted to d+1 dimensions .

The decomposition unit 14 can therefore determine a decomposition into boxes of maximum intersection, and therefore a decomposition into pure boxes of the model , dimension by dimension, by constructing a derived tree structure whose nodes at depth contain the pure regions and therefore correspond to a decomposition into pure regions of leaves restricted to first dimensions. An example of implementation of the method of decomposing the leaves of a model F into pure regions of the “box” type, which can be implemented by the decomposition unit 14, is described in pseudo-code by Algorithm 2 of Annex A1.2. Such a method performs a processing by recurrence, dimension by dimension.

The method of decomposition into pure regions implemented by the decomposition unit 14 (for example according to Algorithm 2 of appendix A.1.2) uses as input the Model of type set of trees defined by:

- N sheets , ..., ;

- Scores associated with each of the sheets with ;

- A vector aggregation function :

The method of decomposition into pure regions implemented by the decomposition unit 14 (for example according to Algorithm 2 of appendix A.1.2) outputs the set pure “box” regions determined for the model and all scores associated with each of the pure regions.

The process of decomposition into pure regions makes it possible to calculate the nodes located at the same depth in a concurrent way.

There illustrates the steps of the method of decomposition into pure boxes (implemented for example according to Algorithm 2 of appendix A.1.2), in an exemplary embodiment.

In the example of the , step E1 corresponds to the input applied to the diagnostic device 100. This input is a collection of multidimensional boxes Bi with their respective scores Si representing the classification model 10. The input query comprises a query point R and a target class C2.

Step E2 corresponds to the pure box decomposition computed recursively through the dimensions, as a so-called “maximum intersection” box decomposition.

Step E3 corresponds to the determination of the scores of the pure boxes obtained by applying the aggregation function "g" of the classification model 10 of the "set of trees" type:

- G1=G2=g(S1);

- G3=g(S1+S2);

- G4=G5=g(S2).

Step E4 corresponds to the determination of the pure boxes belonging to the target class C2 for the counterfactual point associated with R.

Step E5 corresponds to the calculation of the distances from the query point to each of the target boxes.

Step E6 corresponds to the determination of the counterfactual point as the point closest to the query R belonging to the closest target box.

The diagram of the graphically represents an example of a derived tree structure 60 obtained by implementing the method of decomposing the leaves of a model F into pure regions of the “box” type, as implemented by the decomposition units 12 and 14, according to the implementation of appendix A.1 .2 (Algorithm 2) for a simplified F model.

In the example of Figure 6, the model used is a 2-dimensional model (two dimensions d1 and d2) comprising two sheets (F1, F2). The first level 60-1 of the derived tree structure 60 (top right of Figure 6) corresponds to the dimension d1 and to the application of procedure 1 on the leaves restricted to this dimension. The method for calculating the regions of maximum intersection (for example according to Algorithm 1 of appendix A.1.1) is then applied independently in each of the elementary intervals formed in the dimension d1 on the sheets contained in each of these intervals and restricted to dimension d2. Each rectangle with rounded edges 601 is a symbolic representation of an execution of procedure 1 in the dimension associated with the level of the structure (60-1 or 60-2) and on the subset of boxes propagated from the higher level 60-1 and restricted to this dimension (d1 or d2). By following the paths in the formed tree structure 60, a box-like pure region decomposition of the F1 and F2 leaves is determined. The five pure regions extracted are represented by hatched rectangles 602 in Figure 6.

There is a flowchart representing the method of diagnosing abnormalities according to the embodiments of the invention.

At step 700, a request received associated with a target class i is applied to the model F trained beforehand, the request comprising an input datum X and a target class.

At step 702, a decomposition of the classification model into pure boxes is determined, from the input query, said pure boxes corresponding to a collection of intervals, each pure box corresponding to a region of the space of input characteristics, associated with a vector score determined from the scores associated with the leaves of the decision trees (for example according to Algorithm 2 of appendix A1.1).

At step 704, the counterfactual point CF(X,j) representing the virtual point closest to the query point X belonging to the target class is determined, from the decomposition into pure boxes, according to equation 6.

In step 706, the difference between the counterfactual point Y and the input request X is determined, which provides a change vector dX representing the changes to be applied at least to the input data so that the input data is classified in the target class by the classification model, the change vector dX having the same size as the input characteristics.

By definition of the counterfactual point Y, the counterfactual point lies on the outer envelope of a region of the target class j. For any interior point Y of a region R , it is possible to find a point of the envelope of R which is closer to the query X than the point Y. It may suffice for example to select a point which is at inside the segment [X,Y] and at its intersection with the envelope of R.

In one embodiment, step 704 of determining the counterfactual point may include, for each box , determined in decomposition step 702, a determination of the minimum distance from X to the box as well as the point Z of the surface of the box being at a minimum distance between the query point X representing the input data and the pure box, with .

For example, for each box , the point Z can be calculated, by applying a surface point calculation method, such as for example Algorithm 3 described in pseudo-code in appendix A1.3, proceeding dimension by dimension.

In such an embodiment, the point Z located at a minimum distance between the request point representing the input data and the pure box can be determined according to the following equation:

The method for calculating the point Z (such as for example the method implemented by the algorithm of appendix A1.3), for a box representing a D-dimensional box, may comprise a plurality of iterations, each iteration corresponding to a dimension d. At each iteration corresponding to a dimension “d”, the distance corresponding to the minimum distance between X and the box restricted to the first d dimensions is computed, with , Or = .

The distance from a point to a region is defined by equation (5).

The Z point calculation process (for example according to Algorithm 3 of appendix A1.3) uses as input:

- The box ;

- The query point .

The point calculation process outputs the distance from the query point X to the box within the meaning of the definition of equation (5) represented by the point such as :

.

The decomposition step 702 provides a pure box decomposition in the form of a derived tree structure, whose nodes at a depth d contain the regions corresponding to the pure boxes, the depth d corresponding to a dimension of the feature space of entries. Each level of the derived tree structure corresponds to a dimension of the model.

In one embodiment, instead of building the derived tree structure, corresponding to the decomposition into pure boxes, in an exhaustive manner, the construction of the derived tree structure can be stopped at the level of the nodes for which a lower bound is greater than an upper bound , according to a decomposition process with backtracking. The lower bound represents a lower bound on the distance to the query point X of all the pure boxes lying below the node. In one embodiment, for a given node of the derived tree structure, the lower bound is equal to the distance from the query point to the pure box corresponding to this node. The upper bound represents the upper bound on the distance from the counterfactual point to the query point.

Thus, to respond to a query X, instead of determining the decomposition of the model into pure regions of the decision space of the model in an exhaustive way, only a part of the derived tree structure, around the query point, can be built in applying the process of decomposition with backtracking. Such an approach is particularly useful for large F-models for which the number of pure regions explodes with the number of leaves. The property used is the fact that each node of the decomposition tree (derived tree structure) corresponds to a pure decomposition of the leaves indexed by the node and restricted to the current dimension, corresponding to the depth of the node considered in the tree, and that this pure decomposition is compatible with the pure decomposition on all the sheets restricted to this dimension.

According to such a decomposition process with trace feedback, for a query point X, only the nodes of the tree which contain the value of X at the dimension corresponding to the node are constructed. If these nodes are not sufficient to find a pure region of class j , the construction can be extended to neighboring nodes by going up the decomposition structure dimension by dimension.

An example of the method for calculating the upper bound on the distance to the nearest counterfactual point, with backtracking, is provided in pseudo-code in appendix B4.4 (Algorithm 4).

Such a method of calculating the upper bound receives as input the Model F of type set of trees defined by:

- N sheets , ..., ;

- The scores associated with each of the sheets with ;

- The vector aggregation function : ;

The method of calculating the upper bound also receives as input:

- The entry request ; And

- the target class, designated by the “target” variable, with target such as target Class(X).

The upper bound calculation method determines and outputs the upper bound on the quantity dist(X; CF(X; target)).

There is an example of a high-level representation of the operation of the decomposition process with trace feedback corresponding to the application of algorithm 4 of Appendix B.2.4. In the example of the , the target class is class C2. The method generates a construction of a tree structure of the same type as that constructed by the exhaustive decomposition method (for example according to Algorithm 2 of appendix A1.2) in the order indicated by the arrowed path up to that a region belonging to class C2 is determined. The dotted part of the structure is not built. The decomposition process with backtracking applied to a tree structure allows a depth-first traversal as indicated by the arrowed traversal, while the exhaustive decomposition process (according to Algorithm 2 of the Appendix B.2 for example) provides an exhaustive construction of the structure using first a breadth-first traversal.

Once a pure region belonging to the target class is found, the search is reiterated by constructing all the pure regions located at a distance from the query less than or equal to the distance from the query to the pure region found. This last step guarantees to find the closest target pure region.

A method of calculating the counterfactual point CF(X; target) of the "branch-and-bound" type (by separation and evaluation) can then be applied (step 704 of FIG. 7), for example according to Algorithm 5 of the appendix B.2.5 described in pseudo code (the term "target" means "target" in Anglo-Saxon language), using the upper limit calculated . Such a process by separation and evaluation ("branch-and-bound") constructs only a partial decomposition of the decision space of the model located inside a hypersphere of center X and radius . The Ray represents the upper bound on the quantity dist(X; CF(X; target)).

In this embodiment of step 704 of FIG. 4, the method of calculating the counterfactual point CF( , ) receives as input the Model of type set of trees defined by:

- N sheets , ..., ;

- The scores associated with each of the sheets with ;

- The vector aggregation function : ;

- The upper bound on the quantity dist ( ; CF( ; )).

The method of calculating the counterfactual point CF( , target) also receives as input:

- The entry request ; And

- the target class , denoted by the “target” variable in Algorithm 5, with target such as target Class(X), Class( ) designating the class of the input data .

The counterfactual point calculation method CF(X,j) determines and outputs the counterfactual point CF(X,j) in the target class .

Such a method for calculating the counterfactual point CF(X, target) of the "branch-and-bound" type (by separation and evaluation), for example as described in pseudo-code by "Algorithm 5" of appendix B.2.5 , is similar to the process of Algorithm 2 of appendix A1.2 with the exception of the exploration at a node which is abandoned if the distance from the pure region defined by this node to the query (lower bound) exceeds the upper bound calculated by implementing Algorithm 4.

In another embodiment, exploration at a node may not proceed if none of the boxes associated with that node vote for the target region. As target regions are determined inside the hypersphere, the upper bound can be updated with the distance of the query to those target regions.

The anomaly diagnosis device and method according to the embodiments can be used in many applications.

In an exemplary application of the invention, the anomaly diagnosis device and method can be used to detect and diagnose faults in manufacturing processes and production lines. The massive deployment of sensors on modern production lines allows very detailed monitoring of products during manufacture. At the same time, the complexity of the manufacturing processes entails a multiplication of the manufacturing steps, and consequently an increased risk that defects are introduced before the end of the manufacturing process. This also allows the learning of models capable of efficiently detecting defects on manufactured objects. State-of-the-art models do not allow fault identification and diagnosis. However, such information is necessary for maintenance operations on the defective production device(s) in order to prevent the subsequent introduction of defects into the production process by carrying out a timely and targeted maintenance operation.

The anomaly diagnosis device and method according to the embodiments of the invention make it possible to identify and diagnose faults "on the fly" within complex production processes, from the vector dX representing the difference between the counterfactual point and the query point X, the components of the vector dX indicating the minimum changes that should have been made for the input data to be classified in the "normal" class.

The vector dX determined by the anomaly diagnosis device and method provides output data allowing the interpretation of the decision of the binary classification model to place the data characterizing the fault in the "abnormal" class (which corresponds in this example to the "fault" class), and thus a fault diagnosis. By providing a fault diagnosis, the embodiments of the invention allow targeted and rapid intervention on a production line in order to correct the fault and prevent its reintroduction in future production. This results in a saving of time and a minimization of production costs, the production chain no longer having to be completely immobilized for a long period to trace the cause of the defects.

The embodiments of the invention can also be used to detect abnormal behavior (for example the diagnosis of bank or tax fraud), for decision-making (for example the allocation of credits). The embodiments of the invention can also be used in any application requiring not only to classify input data but also to determine data explaining or justifying the decision of the model (for example for constitution of proofs in the case of fraud files, and to justify the decision taken to the applicant in the event of the allocation of credits). In another application, the embodiments of the invention can be used in the medical field for the diagnosis of abnormalities in a patient, such as for example the diagnosis of tumors. In such an application of the invention to the medical field, the decision determined using an artificial intelligence model only has value if it is accompanied by data explaining the decision in order to be able to be validated or invalidated by doctor. It should be emphasized that artificial intelligence models used in the medical field, to be approved, must provide explainable decisions.

For example, in an exemplary application of the invention to obtaining credit in the banking field, a model F of tree set type can be trained beforehand (step 700 of the ) on the allocation of consumer credit. The anomaly diagnosis method can then be applied to diagnose (determine the reason) for each of the negative decisions provided (non-attribution of credit) by the decision model F. The components of the vector dX represent the input characteristics that it would be necessary to change at least for the customer to be eligible (ie for the data to be classified in the "normal" adverse class). Such components can be translated into recommendations, in high-level language.

It is assumed for example that the decision model F is a binary classification model XGBoost comprising 250 decision trees of depth 8 and that the input data set comprises 20 numeric and categorical type input characteristics. An example of a bank credit allocation request is considered. Examples of input characteristics associated with the query include, but are not limited to, number of outstanding loans, repayment history of previous loans, amount borrowed, purpose of loan (type of purchase), salary and the savings of the borrower, the type of employment contract, the duration in the present position held by the borrower, etc. From such input features, the F model is applied, which provides a classification among two classes:

C1: decision to award credit (“normal” class), or

C2: decision to refuse the credit requested by the borrower (“abnormal” class).

Examples of diagnoses of non-attribution of credit formulated automatically from the calculation of the counterfactual point of the “normal” class (which in this case is the class C1 corresponding to a positive decision on the attribution of credit) can include for example the following cases 1, 2 and 3.

Case 1

A request “credit request for $701 made for customer 2” is considered for which the classification given by model F is “credit refused” (C2). The Diagnosis corresponding to the components of the vector dX is the following:
- Entry characteristic 1 corresponding to "Status of existing verification accounts":
Recommended: 0 ≤ ... < 200 – Customer has: < 0
- Entry characteristic 2 corresponding to “Credit History”:
Recommended: all credits on this bank have been duly paid – The status of the Client is: “existing credits duly paid up to now”.
3: Entry characteristic 3 corresponding to “Savings”:
Recommended: 500 ≤ ... < 1000 – Customer has: < 100.
4: Input characteristic 4 corresponding to “Debt ratio as a percentage of income”:
Recommended at most: 2.5 – Customer has: 3
5: Entry characteristic 5 corresponding to “Other debtors/guarantees”:
Recommended: co-applicant – Client status: none.

Case 2

A request “credit request for $330 made for a client 12” is considered for which the classification given by model F is “credit refused” (C2). The Diagnosis corresponding to the components of the vector dX is as follows:
1: Entry characteristic 1 corresponding to “Status of existing checking accounts”:
Recommended: 0 ≤ ... < 200 – Customer has: < 0.
2: Input characteristic 2 corresponding to “Credit period in months”:
Recommended at most: 15 – Client requested: 16.
4: Entry characteristic 4 corresponding to “Current job held since”:
Recommended: 4 ≤ ... < 7 years – Client status is: 1 ≤ ... < 4 years
5: Entry characteristic 5 corresponding to “Other debtors / guarantees”:
Recommended: co-applicant – Client status is: none

Case 3

A request “credit request for $221 made for a client 18” is considered for which the classification given by model F is “credit refused” (C2). The Diagnosis corresponding to the components of the vector dX is as follows:
1: Entry characteristic 1 corresponding to “Status of existing accounts”:
Recommended: 0 ≤ ... < 200 – Client has: < 0.
3: Input characteristic 3 corresponding to “Goal”:
Recommended: car (used) – Customer wants to buy: car (new).
4: Entry characteristic 4 corresponding to “Current job held since”:
Recommended: 4 ≤ ... < 7 years – Client status is: 1 ≤ ... < 4 years.
6: Input characteristic 6 corresponding to “Telephone”:
Recommended: yes, registered under the clients name – Client status: none.

The anomaly diagnosis device and method according to the embodiments of the invention can also be used in model debugging applications (model anomaly diagnosis). Given an input data X of label iet classified by error in the class j by the model (j i), model debugging around the operating point X consists in analyzing the “knowledge” that the model has acquired about the elements of class i close to X, and what additional knowledge would be necessary for the data X to be properly classified in class i by the model. The additional knowledge necessary for the datum X to be suitably classified in class i by the model is represented by the change vector dX calculated by the comparator 18.

The knowledge that the model has of class i can be determined by using elements representative of the decision regions (forming prototypes) associated with class i closest to X.

Model debugging according to such an embodiment can be used to diagnose weaknesses in training models 10 around ambiguous operating points for the model. The diagnosis determined can in particular make it possible to correct the weaknesses of the models by acting on all the learning data, for example by:
- removing adversarial examples that disrupt learning;
- by removing the ambiguities between classes by adding examples in the areas where the model confuses between classes.

The reliability of the diagnosis obtained by exploiting the output (change vector dX) of the anomaly diagnosis device and method can be characterized by the statistical robustness of the diagnosis, that is to say its repeatability either on similar models, or on similar input data.

To evaluate the repeatability of the diagnosis on similar models, instead of training several different models, it is possible to simulate close models by randomly ignoring pure decision regions of the model. In practice, by using the method of determining the counterfactual point (for example according to algorithm 5 of appendix A1.5), it is possible to answer the query: what are the M closest counterfactual examples belonging to M distinct pure regions. By analyzing the statistical invariants inside the set , it is then possible to extract a robustness measure of the diagnosis.

To assess diagnostic repeatability on similar input data, considering the neighbors of inside a radius ball and the difference at their respective counterfactual points it is possible to characterize the stability/repeatability of the invariant. It is then possible to apply diagnostic robustness characterization techniques, for example as described in D. Alvarez-Melis and TS Jaakkola, “on the robustness of interpretability methods,” arXiv preprint arXiv:1806.08049, 2018.

The anomaly diagnosis device and method 100 according to the embodiments of the invention can be used in different applications, for example for fault diagnosis. It allows the analysis of relatively small changes in the input characteristics (given by the vector dX) so that it is possible to determine very precisely the origin of the anomaly (defect for example) while providing quantitative information on the “magnitude” of the observed changes.

The anomaly diagnosis device and method 100 according to the embodiments of the invention advantageously exploit the particularity of models of the "set of decision trees" type of constructing in an explicit, exhaustive and exact manner the decision space of these models.

The embodiments of the invention have several advantages, including:
- they allow an interpretation of the decision of the model at a given point;
- they are specific to the decision model;
- they do not present inaccuracies and instabilities;
- they are non-parametric;
- they are adaptable to arbitrary model sizes;
- they apply to a wide variety of models (XGBoost, LightGBM, ...) which are commonly used for anomaly diagnosis, whatever the nature of the input characteristics (numerical and/or categorical industrial data) ;
- the determination of the pure decision regions of the model 10 can be used in many other applications, for example, for model knowledge prototyping or model debugging.

The diagnostic device 100 can be implemented on a distributed computing unit comprising a plurality of physical cores (for example 16 cores). The embodiments of the invention are particularly suitable for such a distributed implementation as well as for porting to multi-core hardware technologies, which makes it possible to obtain a rapid response to a user request regardless of the size of the model. of classification considered. The diagnostic device 100 can be implemented in different languages, such as the C++ language.

More generally, those skilled in the art will understand that the diagnostic device 100 or subsystems of the device according to the embodiments of the invention can be implemented in various ways by hardware (“hardware”), software, or a combination of hardware and software, especially in the form of program code that can be distributed as a program product, in various forms. In particular, the program code may be distributed using computer readable media, which may include computer readable storage media and communication media. The methods described in this description may in particular be implemented in the form of computer program instructions executable by one or more processors in a computer computing device. These computer program instructions may also be stored in computer-readable media.

Furthermore, the invention is not limited to the embodiments described above by way of non-limiting example. It encompasses all the variant embodiments which may be envisaged by those skilled in the art. In particular, the invention is not limited to the examples of application described above such as the diagnosis and the diagnosis of faults. The embodiments of the invention can be applied, for example, to diagnosis in medical Artificial Intelligence, to the diagnosis of fraud, to the analysis of multi-class decision-making in general.

ANNEX A

A.1.1. Algorithm 1

A.1.2. Algorithm 2

A.1.3. Algorithm 3

A.1.4. Algorithm 4

A.1.5. Algorithm 5

Claims (13)

1. Anomaly diagnostic device (100) comprising:
- a classification unit (11) configured to determine, in response to a request received comprising an input datum and a target class, the class to which the input datum belongs among a set of classes, by applying the datum d input to a multiclass classification model, the input data corresponding to a query point,
said classification model (10) using a set of decision trees, the input data comprising a set of input features, the space of input features having a given number of dimensions,
each decision tree comprising a set of nodes defining a tree structure from a root node to a set of leaves, each leaf of the decision tree comprising a set of scores associated with each class, each decision tree being applied to the input datum for determining an elementary classification decision from the scores associated with the leaves of the decision tree, said classification unit (11) applying an aggregation function to the elementary classification decisions determined by each of the trees of the model to determine the class of the input data;
- a decomposition unit (14) configured to decompose at least part of the set of leaves associated with the decision trees of the model into multidimensional boxes, called pure boxes, from the input request, said pure boxes corresponding to a collection of intervals, each pure box corresponding to a region of the input feature space, associated with a vector score determined from the scores associated with the leaves of said decision trees;
- a counterfactual point calculation unit (17) configured to calculate a counterfactual point (Y) representing the virtual point closest to the query point belonging to the target class, from the decomposition into pure boxes;
- a change vector calculation unit (18) configured to compare the counterfactual point with the input query and determine the difference between the counterfactual point (Y) and the input query (X), thereby providing a vector change vector (dX) representing the minimum changes to be applied to the input data for the input data to be classified in the target class by the classification model, the change vector (dX) having the same size as the input characteristics. 2. Device according to claim 1, characterized in that the decomposition unit (14) determines the decomposition into pure boxes in the form of a derived tree structure, whose nodes at a depth d contain the regions corresponding to the pure boxes , the depth d corresponding to a dimension of the input feature space. 3. Device according to one of the preceding claims, characterized in that the decision trees of the model comprise intersecting leaves, and in that the decomposition unit (14) is further configured to determine the decomposition of the model in pure boxes in the form of regions of maximum intersection, called maximum intersection boxes, a maximum intersection box representing a region of the model having a uniform score over the whole of the region resulting from the aggregation of the scores of the sheets which intersect to form said region, by applying said aggregation function. 4. Device according to claim 3, characterized in that the decomposition unit (14) is configured to determine the maximum intersection boxes of the model by induction on the dimensions of the space of the input characteristics, by applying a plurality of iterations, each iteration corresponding to one of said dimensions, the decomposition unit (14) supplying the decomposition into pure boxes of the model in the form of boxes of maximum intersection dimension by dimension. 5. Device according to one of Claims 3 and 4, characterized in that the decomposition unit (14) is configured to determine, from a decomposition into maximum intersection boxes of the restriction of the model (10) to the first d dimensions of the model, a decomposition into maximum intersection boxes of the restriction of the model to the first d+1 dimensions of the model, d denoting a given dimension of the model, a restriction of the model to the first d dimensions denoting the restriction of the set of boxes corresponding to the leaves of the decision trees of the model at the first d dimensions of the model. 6. Device according to one of claims 4 to 5, characterized in that the maximum intersection boxes are maximum intersection boxes associated with the restriction of the model to the first d+1 dimensions. 7. Device according to one of the preceding claims, characterized in that the model is broken down into a set of pure boxes comprising at most elements, N denoting the number of leaves of the decision trees of the model (10) and representing the dimension of the input feature space applied to the model (10). 8. Device according to one of the preceding claims, characterized in that the unit for calculating the counterfactual point is configured to calculate the counterfactual point, by determining, for each pure box, a point on the surface of the pure box located at a minimum distance between the query point representing the input data and the pure box. 9. Device according to claim 8, characterized in that the point Z of the surface of the pure box Bi located at a minimum distance between the query point X representing the input data and the pure box Bi is given by:
10. Device according to one of the preceding claims, characterized in that the device is configured to calculate a lower bound and an upper bound for each node of said derived tree structure corresponding to the decomposition into pure boxes, each level of the tree structure derivative corresponding to a dimension of the model, the lower bound representing a lower bound on the distance to the query point X of all the pure boxes located below said node, the upper bound representing the upper bound on the distance from the counterfactual point to the query point , the device being configured to stop the construction of said derived tree structure at the nodes for which the lower bound is greater than the upper bound. 11. Device according to claim 10, characterized in that for a given node of the derived tree structure, the lower bound is equal to the distance from the query point to the pure box corresponding to this node. 12. Device according to claim 11, characterized in that the counterfactual point for a query point X and a target class "j" is determined from the pure boxes located at depth D of the tree structure calculated for the model according to the equation:
13. Anomaly diagnosis method (100) comprising the steps of:
- in response to a request received comprising an input datum and a target class, determining the class to which the input datum belongs among a set of classes, by applying the input datum to a multiclass classification model, the input data corresponding to a query point,
said classification model (10) using a set of decision trees, the input data comprising a set of input features, the input feature space having a given number of dimensions, each decision tree comprising a set of nodes defining a tree structure from a root node to a set of leaves, each leaf of the decision tree comprising a set of scores associated with each class, each decision tree being applied to the input data for determining an elementary classification decision from the scores associated with the leaves of the decision tree, an aggregation function being applied to the elementary classification decisions provided by each of the trees of the model to determine the class of the input data;
- decomposing at least part of the set of leaves associated with the decision trees of the model into multidimensional boxes, called pure boxes, from the input query, said pure boxes corresponding to a collection of intervals, each pure box corresponding to a region of the input feature space, associated with a vector score determined from the scores associated with the leaves of said decision trees;
- calculate a counterfactual point (Y) representing the virtual point closest to the query point belonging to the target class, from the decomposition into pure boxes;
- determine the difference between the counterfactual point (Y) and the input query (X), which provides a change vector (dX) representing the changes to be applied at least to the input data so that the input is classified into the target class by the classification model, with the change vector (dX) having the same size as the input features.