EA040190B1 - METHOD OF INTERPRETATION OF ARTIFICIAL NEURAL NETWORKS - Google Patents

Description

Technical field

The invention, in general, relates to the field of computer technology, and in particular to methods and systems for interpreting the operation of artificial neural network models.

State of the art

Currently, artificial neural networks are an important tool for solving many applied problems. They have already made it possible to cope with a number of difficult problems and promise the creation of new inventions capable of solving problems that so far only man can do. Artificial neural networks, as well as biological ones, are systems consisting of a huge number of functioning neuron processors, each of which performs a small amount of work assigned to it, while having a large number of connections with the rest, which characterizes the strength network computing.

One of the disadvantages of artificial neural networks is the complexity of meaningful interpretation, including the difficulty of substantiating the activation of neurons. Interpretability problems lead to a decrease in the value of the results of the work of an artificial neural network. The internal representation of learning outcomes is often so complex that it cannot be analyzed even by an expert in the prior art, except in some simple cases that are usually of no interest.

Currently, artificial neural networks are used in many areas of technology, but before they can be used where human lives or significant material resources can be endangered, important issues regarding the reliability of their work must be resolved, and therefore the interpretation artificial neural networks takes on additional importance.

The essence of the invention

The invention is aimed at eliminating the disadvantages inherent in existing solutions from the prior art.

The technical problem posed in this technical solution is to represent the rules for making a decision by a neural network in the form of easily interpreted logical expressions.

The technical result achieved by solving the above problem is to improve the quality and accuracy of the interpretation of the artificial neural network.

The specified technical result is achieved due to the implementation of a method for interpreting artificial neural networks, in which at least one artificial neural network pre-trained on a set of objects is obtained; further, at least one decision tree is formed for each layer of the trained neural network, and the decision tree receives as input the activation of the corresponding layer obtained when passing through the neural network of the object, from the available data set; predicting, by means of decision trees, the answer that the trained artificial neural network produces at this object; then, for each object, an ordered sequence of leaf numbers of the decision trees formed at the previous step is obtained; after that, a set of rules is formed that predicts the sequence of leaf numbers for the object.

In some embodiments of the invention, the cross entropy between the class distribution predicted by it and the distribution returned by the classifier is used as a measure of the quality of the decision tree.

In some embodiments of the invention, the mean square or absolute error between the answer predicted by it and the answer returned by the classifier is used as a measure of the quality of the decision tree.

In some embodiments of the invention, decision trees are built independently of each other.

In some embodiments of the invention, the decision trees are built dependent on each other based on the boosting algorithm.

In some embodiments of the invention, decision trees are built dependently on each other by adding information about the numbers of leaves from trees built on previous layers to the input of the tree of the next layer.

In some embodiments, XGBoost or AdaBoost or LPBoost or TotalBoost or BrownBoost or MadaBoost or LogitBoost is used as the boosting algorithm.

In some embodiments of the invention, a decision tree is formed for each layer of the trained neural network based on the CLS algorithm, or ID3, or C4.5, or CART, or IndCART, or DB-CART, or CHAID, or MARS.

In some embodiments of the invention, a decision tree is formed based on the minimization of the error functional for all objects that are fed to it as input.

In some embodiments of the invention, the input to the decision tree is the original feature description for an artificial neural network or a description of an object with a modified set of features.

- 1 040190

In some embodiments, the leaves in each decision tree are numbered.

In some embodiments of the invention, an ordered sequence of leaf numbers is formed by the layer number in the neural network.

Brief description of the drawings

The features and advantages of the invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which: FIG. 1 is a block diagram of a method for interpreting artificial neural networks;

in fig. 2 shows a variant of the implementation of getting an object into the decision tree when the object gets into one of the leaves of the tree;

in fig. 3 - an implementation option for the formation of decision trees for each layer of the ANN and the resulting discrete trajectories along the leaves of the decision trees where the object falls;

in fig. 4 - an implementation option for the formation of a decision tree that predicts the sequence of leaf numbers for the object;

in fig. 5 - an example of implementation, according to which the minimum number of constructed discrete trajectories is selected from the general set, which cover at least 90% of the examples;

in fig. 6 - averaging objects in different discrete trajectories, leading to a high probability of symbol 2;

in fig. 7 - an example of the implementation of images of a discrete trajectory with only five examples of different spellings of the symbol 3;

in fig. 8 is a variant of the resulting discrete trajectory leading to a high probability of symbol 7, and the binarized and weighted rules leading to this trajectory.

Detailed description of the invention

Concepts and terms necessary for understanding the invention will be described below.

In this invention, the system means, among other things, a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of operations (actions, instructions).

A command processing device is an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. The role of a storage device can be, but not limited to, hard drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

Program - a sequence of instructions intended for execution by a computer control device or a command processing device.

An artificial neural network (hereinafter - ANN) is a computational or logical circuit built from homogeneous processor elements, which are simplified functional models of neurons.

A decision tree is a graph, a diagram that reflects the structure of an optimization problem for a multi-step decision-making process.

It is used in classification problems and in other areas for analyzing solutions, structuring problems. The branches of the tree display various events that may take place, and the nodes (vertices) of the states in which the need for selection arises. In this technical solution, this is a way of dividing the space, since each condition divides the space with a hyperplane perpendicular to the axis along which the choice is made.

Boosting is a machine learning meta-algorithm for performing supervised learning.

A fully connected neural network is a neural network in which each neuron transmits its output signal to the input of the neurons of the next layer.

Neural network layer (English layer) - a set of network neurons, combined according to the features of their functioning.

The activation function of a neural network is a function that is used to convert the activation level of an element (neuron) into an output signal.

Typically, the activation function has a squeezing action.

A predicate is an element of an attributive judgment denoting some sign (property) of its subject, or what is said about the subject.

Tuple - in mathematics, a sequence of a finite number of elements.

For example, a graph is defined as a tuple (V,E), where V is a set of vertices and E is a subset of V x V denoting edges. In set theory, a tuple is usually defined inductively.

The method for interpreting artificial neural networks shown in FIG. 1 in flowchart form may include the following steps.

The task is to classify and regress a data set using an artificial neural network and then analyze the resulting network in order to find rules that characterize the process.

- 2 040190 transformation of the input data by the network to obtain a response.

Step 110: at least one artificial neural network pre-trained on a set of objects is obtained.

Let in this technical solution get a pre-trained neural network with D layers (each layer contains Si neurons) with layer sizes S1,...,S _D , and a training sample •

At this step, additional data is collected for further processing, namely, for each layer of the neural network d = 1, ..., D, all neuron outputs (defined as activations) are obtained for each training set of objects as D matrices Ad of size N x S _d . Also, the results of the obtained class probabilities are formed as an N x C matrix P, where C is the output dimension (the number of classes or the dimension of the regression variables).

The number of artificial neural network layers is not limited by implementation options. As a trained neural network, a fully connected neural network, or a convolutional neural network, or a recurrent neural network, or a combination of them can be used, without being limited. As an activation function, a rectified linear unit (ReLU) can be used, which is expressed by the following formula:

In some implementations, the Sigmoid, tanh, LeakyReLU, PReLU, and other activation functions known in the art may be used, without being limited.

An artificial neural network can be trained on a set of objects, which, for example, are images. The training sample can consist of positive and negative examples. In some implementations, the training sample may consist of examples of an arbitrary number of classes, may not be labeled at all, or may be a regression problem, where classes are an inapplicable concept. For example, if an ANN is trained to detect human faces, the training set consists of images that contain faces and images that do not have faces. The ratio of positive examples to negative examples can be chosen as N:M, eg 4:1, eg 8000 positive and 2000 negative.

In a specific implementation example, the LFW3D database can be used as a positive training sample. It contains 13000 color images of frontal faces of the JPEG type, 90x90 pixels in size.

As negative training examples, the SUN397 database can be used for training, which contains a huge number of various scenes, which are categorized. In total, this database contains 130,000 images, 908 scenes, 313,000 scene objects. The total weight of this database is 37 GB.

In the case of using a fully connected or convolutional neural network and when the objects are handwritten digits, the MNIST database can be used for training, namely a large database of handwritten digit samples.

An artificial neural network is transmitted and stored as architecture and weight values in its layers. In some implementations, the architecture of the network and the weights in the layers are stored separately so that the weights can be loaded into a network with a different architecture. This approach is used, for example, when combining learning without a teacher and with a teacher. The first step is unsupervised learning using an autoencoder, deep web of trust, or other method. Then the obtained weights are loaded into a network of a different architecture, which is further trained with a standard supervised learning approach using the backpropagation method. Combining the two methods allows you to train the network in the case when there is little labeled data for training. The weight values in the layers can be stored, for example, in the HDF5 data format. The contents of HDF5 files are organized like a hierarchical file system, and data is accessed using paths similar to POSIX syntax, such as /path/to/resource. Metadata is stored as a set of named object attributes.

In some implementations, feature set normalization is performed for ANNs because neural networks work better with data represented by numbers normally distributed around 0, and the original data may have an arbitrary range or not be numeric data at all. In this case, various methods are possible, ranging from a simple linear transformation to the required range and ending with a multivariate analysis of the parameters, and nonlinear normalization, depending on the influence of the parameters on each other.

Step 120: At least one decision tree is generated for each layer of the trained neural network.

The decision tree consists of two types of vertices, as shown in Fig. 2. Vertices of solutions containing conditions are denoted by circles. Goals or inferences are indicated by boxes. Vertices are numbered and conditions are set on arcs. Each vertex can have at most one input. The paths of movement along the tree from the top level to the lowest ones define logical rules in the form of chains of conjunctions. The rules expressing regularities are formulated as productions: IF A THEN B or in the case of a set of conditions:

IF (condition 1) ^Λ (condition 2) ^Λ ... ^Λ (condition N) THEN (value of output node).

The formation of decision trees can be based on expert assessments or using example processing algorithms (CLS, ID3 - Interactive Dichotomize ^ C4.5, CART - classification and regression trees, IndCART, DB-CART, Chi-square automatic interaction detector (CHAID), MARS, but not limited to, etc.).

In one implementation, the decision trees for each layer of the neural network can be built independently of each other. A decision tree is a list of conjunctions as shown in Fig. 2, where at each vertex of the decision tree there are predicates or rules (shown as x and y), which include two arguments: a specific feature of the object that is input to the decision tree, as well as a particular value of this feature. The features of the object are directly fed into the decision tree only at the zero (input) layer. On all other layers, decision trees accept activations received when an object passes through the corresponding layer. An activation function is a function that takes a weighted sum as an argument. The value of this function is the output of the neuron in the ANN.

At this step, D independent decision trees T _d (for predicting finite probabilities P from A _d ) are obtained, trained on matrices A _d as input and finite probabilities P as output. This implementation can be thought of as a special type of multi-tree creation with tree averaging, where each decision tree is built on separate inputs. If the predicate is fulfilled for a specific given value of an object attribute in this decision tree, the tree object falls into the right or left branch, depending on the fulfillment of the predicate condition. The decision tree is built on the basis of minimizing the error functional for all objects that are fed to it as input. The goal of a decision tree is to make the most accurate prediction possible. To estimate the accuracy, the error functional is used (or, if we take it with the opposite sign, the quality functional.) The lower the values of the error functional, the better the decision tree solves the problem. Each new condition at the top of the decision tree is chosen so that as a result, the overall error functional is reduced as much as possible.

In some embodiments, all the features of all objects of the training sample for the ANN take part in the decision tree formation procedure. The input features for the first decision tree on the zero layer are features of objects, and for the next decision tree, the values of activations of the corresponding layer on the sample objects are used. In some implementations, regression decision trees are formed that do not predict the probability of a particular classification in the last layer of the ANN, but predict real values (for example, class scores) that can be uniquely converted to probability values. At the same time, the reverse conclusion is not true, since different real values can correspond to the same probabilities at the ANN output. Several different sets of such values, when applying a deterministic transformation procedure, can give the same probability value at the last layer of the ANN.

In another embodiment, the decision trees are not built independently of each other, but dependently based on the boosting method. Boosting is a procedure for sequential construction of the composition of machine learning algorithms, when each next algorithm seeks to compensate for the shortcomings of the composition of all previous algorithms. Boosting is a greedy algorithm for constructing a composition of algorithms. In this embodiment, decision trees are sequentially formed, with each of the subsequent trees adding something to the ensemble, correcting or improving the previous result. Depending on the number of the tree that is being formed at a given time, the data on which this tree is built also changes. That is, if an ensemble of decision trees up to the Nth layer is formed, then the (N + 1)-e decision tree is formed on activations of the (N + 1)-th ANN layer, thus, each decision tree is assigned to the ANN layer. In a specific implementation, the first model is formed (meaning the decision tree), then the second, which tries to correct the errors of the first, then the third, which tries to correct the error of the first two, and so on. In this case, the error correction usually occurs as follows: the error functional of an already prepared composition of decision trees is taken, its gradient is calculated - that is, the direction of the fastest increase in the error, and the next model tries to approximate this gradient, so that then its prediction, being subtracted, would be directed to the side the fastest decrease in error. At this step, D - 1 trees are formed by boosting. The training takes place in the following way. Select AD scores as the target for evaluation. A base estimate is formed that predicts the average scores from all training sets and this estimate is taken as the current best estimate C. For each layer d = 1, ..., D - 1, a new decision tree is formed in the following steps:

a) prepare a new target, like μ = AD - C, to allow the next evaluation to correct the error of the previous one;

- 4 040190

b) form a decision tree Td that takes Ad as input and outputs μ as the target;

c) collecting a training set of current tree predictions as a prediction;

d) choose the coefficient β by minimizing the cross entropy functional (Softmax (C β * prediction), P) over β;

e) choose a new C^C - β * prediction.

Thus, these decision trees can be thought of as a special case of the boosting algorithm, where different data is used for each new tree, and β is chosen with a different goal from the decision tree. This strategy is that in this implementation there are not many independent trees, but trees that try to get new information from each layer, not forgetting previous partitions.

In some implementations, the decision tree boosting algorithm uses the same set of features to build each decision tree, from which a random subsample is made.

The boosting algorithm can be XGBoost, AdaBoost, LPBoost, TotalBoost, BrownBoost, MadaBoost, LogitBoost, but not limited to.

In yet another implementation, the following algorithm is used to generate decision trees. The features of each object of the training sample for the i-th tree are replenished with the number of the leaf in which it fell in the previous i - 1 tree or in all previous trees from 1 to i - 1.

Step 120.1: The decision tree receives as input the activations of the corresponding layer, obtained by passing through the neural network of the object, from the available set.

Step 120.2: using decision trees, predict the same answer that the trained artificial neural network gives on this object.

An ensemble of decision trees distills the response of the neural network, i.e. form the same answers by means of a decision tree on the objects that the network gives them.

The deeper the decision trees, the greater the number of potentially possible discrete trajectories, which is bad. However, the less deep the decision trees, the worse the quality of the prediction of the final ensemble of decisions.

Step 130: for each object, an ordered sequence of leaf numbers from the decision trees formed in the previous step is obtained.

An ordered sequence of decision tree leaf numbers is formed by the layer number in the neural network. This means that if in the zero decision tree the object was in leaf a, in the first one - in leaf b, and in the third - leaf c, then as a result this ordered sequence of leaf numbers is reduced to a tuple a, b, c. A necessary condition is that the order is the same for all objects and in some embodiments of the invention may have a different principle of formation.

So, for one input X, a discrete set of numbers (a discrete trajectory) is obtained as follows:

a) get the corresponding activations a _b ..., a _D ;

b) Run activations through linked trees T and collect predictive leaf IDs L = L1, ..., L _D .

After forming a set of activation trajectories as Cartesian products of decision tree leaf indicators, the most interesting ones can be selected. In some implementations, the leaves in each decision tree are numbered so that they can be uniquely referenced. In fact, a discrete trajectory is a tuple, where there are leaves of trees in a row, into which the object fell through the ANN layers. For example, there are two decision trees, where the first decision tree has five leaves and the second has five leaves, which are formed for different layers, so that the total number of potential trajectories reaches twenty-five.

It turns out a discrete version of the trajectory along which the object passed as a set of its features that are fed to the input of the ANN when the neurons of the ANN are activated.

Step 140: a set of rules is formed that predicts the sequence of leaf numbers by object.

Based on the entire ensemble of decision trees for objects and their activations, they analyze what the ANN does with the object in a particular layer. At this step, it is necessary to understand by what features of the initial description the neural network determines the object into one or another discrete trajectory. Thus, the problem of prediction is solved using a set of rules for the number of a discrete trajectory based on the initial set of interpreted features of the object. In other implementations, the set of interpreted features is not the original one, but is built on the basis of the original features. It is necessary to define the rules that describe which trajectory will fall into which object, i.e. discrete trajectory number. For example, when analyzing a text, the entire sample of the text itself is fed into the neural network. As the mentioned transformation, the formation of a bag of words can serve, when it is recorded in the data warehouse, how many times each word was encountered in the text, when

- 5 040190 In this case, all words are reduced to their initial form, as a result of which a digital description of the text appears. This transformation is uniquely performed in one direction (from the text to the bag of words) and ambiguously in the opposite direction, since it is impossible to uniquely restore the original text from the bag of words.

The initial feature space, on which the ANN was trained, is fed to the input of a set of rules. A decision tree can be used as an algorithm for constructing rules. In some embodiments, the input to the decision tree is not the original feature description, but a description of the given object with a modified set of features. This set can be modified to help build expressive rules. The rules will look like IF interpreted_feature_1 < value_1 AND ... AND interpreted_feature_n < value_n, THEN the object will follow a discrete trajectory x. For example, ANNs are trained in advance on photographs of people, and the decision tree is fed into the input hair color, distance between pupils, etc., formed in the form of a table. In another example, a neural network is trained on a sequence of user transactions, however, this data set is not interpretable. The modified interpretable attribute will be the following: what proportion of the amount the user spends on Mondays, Tuesdays, Wednesdays, etc. Thus, the decision tree is already built on a set of modified features.

In some embodiments, a coefficient is generated for each item that determines its importance. During the procedure for constructing the final rules, different examples may have different importance, that is, the degree of influence on the resulting rules. For example, the coefficient can be formed based on how much the prediction of an artificial neural network on an object differs from the prediction of an ensemble of layered decision trees.

As a global criterion for completing the work of a technical solution, the maximum size of the tree and the overall assessment of the quality of the classification of examples by the tree can be used. Of course, the deeper the tree (the longer the set of rules), the more accurately it will predict the discrete trajectory. But too many rules complicate interpretation. After building the decision tree, it is also necessary to post-process the rules according to one of the well-known algorithms. In some implementations, the construction of a set of rules can also occur directly, without the participation of a decision tree, for example, through the RuleFit algorithm.

The main advantage of this approach is the discretization of the ANN activation sequence, which makes it possible to obtain an interpretation using rules. Moreover, the server includes various hardware components, including one or more single or multi-core processors, which are represented by a processor, a graphics processing unit (GPU), a solid state drive, a RAM, a monitor interface, and an I/O interface.

Communication between the various server components may be via one or more internal and/or external buses (e.g., PCI bus, Universal Serial Bus, IEEE 1394 high speed bus, SCSI bus, Serial ATA bus, etc.) to which various hardware components. The monitor interface can be connected to a monitor (e.g. via HDMI cable) visible to the operator, the I/O interface can be connected to a touch screen, keyboard (e.g. via USB cable) and mouse (e.g. via USB cable), both keyboard and mouse are used by the operator. In accordance with embodiments of the present technology, a solid state drive stores program instructions suitable for loading into RAM and used by the processor and/or GPU to select a given process feature target from a set of features and a given type of output values, as will be described below. For example, the program instructions may be part of a library or an application.

The server may be a desktop computer, laptop, tablet, smartphone, personal digital organizer (PDA), or other device that may be configured to implement the present technology, as will be understood by one skilled in the art. The server may be configured to implement a machine learning algorithm (MLA) and perform various methods for learning the MLA. In some embodiments of the present technology, the MLA may be either an artificial neural network, Bayesian network, support vector machine, etc. In another embodiment of the present technology, the MLA may be a prediction model that includes a set of decision trees for solving regression and classification problems, among other things. In this case, MLA can be trained using machine learning techniques such as gradient boosting.

The server may be configured to perform a plurality of procedures, wherein at least one of the plurality of procedures is to create a training sample for training the MLA. In general, the MLA can be trained to predict the design errors inherent in the calculation methods. How a server can be configured to create a training sample for MLA training will be described below.

In some embodiments of the present technology, the server may be configured to access history data associated with a user's financial transactions.

- 6 040190 or other data. History data can be stored locally on the server's SSD. In other embodiments of the present technology, history data may be stored remotely on a storage medium that is operatively connected to a server over a network. In this case, the server can retrieve the history data from the storage medium over the network. The main advantage of this technical solution lies in the generalizing ability of artificial neural networks, which makes it possible to obtain simpler decision trees, i.e. the so-called distillation is carried out. Distillation is not carried out in the standard and known from the prior art, but allows you to determine which objects the ANN tends to process in a similar way, i.e. what patterns in the data are distinguished by the existing trained neural network. Thus, the invention makes it possible to extract structured knowledge not only from extremely simplified neural networks, but also from neural networks that are interpreted in finished form, without simplifying their structure during the analysis, which makes it possible to use it in a wide range of practical problems.

The modules described above and used in this technical solution can be implemented using electronic components used to create digital integrated circuits. Not limited to, microcircuits, the logic of which is determined during manufacture, or programmable logic integrated circuits (FPGA), the logic of which is set by programming, can be used. Programmers and debugging environments are used for programming, allowing you to set the desired structure of a digital device in the form of a circuit diagram or a program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs are programmable logic controllers (PLCs), basic matrix crystals (BMC ) requiring a factory manufacturing process for programming; ASICs are specialized custom large integrated circuits (LSI), which are significantly more expensive in small-scale and single-piece production.

Also, modules can be implemented using read-only memory devices (see Lebedev O.N. Memory microcircuits and their application. - M .: Radio and communication, 1990. - 160 s; Large integrated memory circuits: Handbook / A.Yu. Gordenov and others - M .: Radio and communication, 1990. - 288 p.). Thus, the implementation of all used blocks is achieved by standard means based on the classical principles of implementing the fundamentals of computer technology, known from the prior art.

Implementation examples

The invention can be implemented using MNIST data sampling and a fully connected ANN.

The network model will be explained below. In this technical solution, a feed-forward network was trained, configured as follows: (784, 200, ReLU) - (20o, 200, ReLU) - (200, 200, ReLU) - (200, 10, LogSoftmax). The data triple (784, 200, ReLU) implies that there is a layer that has an input dimension of 784, an output dimension of 200, and an activation function of ReLU. The model was trained with the Adam optimizer with a training step of 3e-4. The network was trained for 10 epochs and achieved 97.5% accuracy on the test set. Activations were obtained after each use of the ReLU activation function, as well as before the LogSoftmax function (evaluation for each class), thus obtaining three activation sets of 200 values and one activation set of 10 values.

Further, in a specific embodiment, the strategy of forming decision trees through boosting is used. Then we limited the maximum depth of decision trees to three and the minimum number of objects in a leaf - 3% of the sample size. The Scikit-learn library is used as decision trees. Class grades were chosen as the target for boosting. The cross entropy of the ensemble of decision trees turned out to be 0.54. Features with the top 5% boost error were excluded from the final tree training set. Next, a minimum number of discrete trajectories were selected that included at least 90% of the training sample examples, as shown in FIG. 5. Selected discrete trajectories retain their original ID as long as all other trajectories are flagged as -1. Various discrete trajectories leading to the same number are shown in FIG. 6.

Then the final decision tree is formed, in which the original image is the input, and the number of the discrete trajectory is the target for determination. Thus, we have a problem with several N + 1 discrete trajectories, where N discrete trajectories from the previous step are selected and 1 trajectory contains all other samples. In FIG. 8 shows a discrete trajectory that belongs to symbol 7 with a high probability and the rules that define this trajectory. In the binary image rules, pixels are red (not shown in the b/w version of the figure), if it was included in the final decision tree with a negative value (the pixel value must be lower than the threshold set in the tree node), green are pixels with positive value, otherwise - yellow (colors are not shown in the b/w version). The weighted part tells each pixel about a brightness value proportional to the number of instances affected by this rule. For a person skilled in the art it is obvious that in the present description, the expression receiving data from the user means receiving an electron

- 7 040190 data device from a computer system, server, etc. in the form of an electronic (or other) signal. In addition, those skilled in the art will appreciate that displaying data to a user via a graphical user interface (e.g., a screen of an electronic device, and the like) may include transmitting a signal to the graphical user interface, that signal contains data that can be processed, and at least some of this data may be displayed to the user via a graphical user interface. Some of these steps, as well as transmitting/receiving a signal, are well known in the art and have therefore been omitted from specific parts of this specification for simplicity. Signals can be transmitted/received by optical means (eg, fiber optic connection), electronic means (eg, wired or wireless connection), and mechanical means (eg, based on pressure, temperature, or other suitable parameter).

Modifications and improvements to the above described embodiments of the present technology will be apparent to those skilled in the art. The foregoing description is provided by way of example only and is not intended to be limiting in any way. Thus, the scope of the present technology is limited only by the scope of the appended claims.

Claims (12)

CLAIM 1. A method for interpreting artificial neural networks, including the following steps: receive at least one artificial neural network pre-trained on a set of objects; at least one decision tree is formed for each layer of the trained neural network, wherein the decision tree receives as input the activation data of the neurons of the corresponding layer obtained when passing through the neural network of the object from the existing data set; predicting, by means of decision trees formed for each layer of the trained neural network, the same answer that the trained artificial neural network produces at this object; get for each object an ordered sequence of leaf numbers generated at the previous step of the decision trees; based on the ordered sequence of leaf numbers and neuron activation data, a set of rules is formed that predicts the sequence of leaf numbers for the object. 2. The method according to claim 1, characterized in that, as a measure of the quality of the decision tree, the cross entropy between the class distribution predicted by it and the distribution returned by the ANN is used. 3. The method according to claim 1, characterized in that, as a measure of the quality of the decision tree, the mean square or absolute error between the answer predicted by it and the answer returned by the ANN is used. 4. The method according to claim 1, characterized in that the decision trees are built independently of each other. 5. The method according to claim 1, characterized in that the decision trees are built depending on each other based on the boosting algorithm. 6. The method according to claim 1, characterized in that the decision trees are built depending on each other by adding information about the numbers of leaves from trees built on previous layers to the input of the tree of the next layer. 7. The method according to claims 5 and 6, characterized in that XGBoost, or AdaBoost, or LPBoost, or TotalBoost, or BrownBoost, or MadaBoost, or LogitBoost is used as the boosting algorithm. 8. The method according to claim 1, characterized in that a decision tree is formed for each layer of the trained neural network based on the CLS algorithm, or ID3, or C4.5, or CART, or IndCART, or DBCART, or CHAID, or MARS. 9. The method according to claim 1, characterized in that the decision tree is formed based on the minimization of the error functional for all objects that are fed to it as input. 10. The method according to claim 1, characterized in that the input to the decision tree is an initial feature description for an artificial neural network or a description of an object with a modified set of features. 11. The method according to claim 1, characterized in that the leaves in each decision tree are numbered. 12. The method according to claim 1, characterized in that the ordered sequence of leaf numbers is formed according to the layer number in the neural network.