EP3782079A1 - Interprétation de modèle - Google Patents

Interprétation de modèle

Info

Publication number
EP3782079A1
EP3782079A1 EP19788527.0A EP19788527A EP3782079A1 EP 3782079 A1 EP3782079 A1 EP 3782079A1 EP 19788527 A EP19788527 A EP 19788527A EP 3782079 A1 EP3782079 A1 EP 3782079A1
Authority
EP
European Patent Office
Prior art keywords
model
linear
feature
machine learning
surrogate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19788527.0A
Other languages
German (de)
English (en)
Other versions
EP3782079A4 (fr
Inventor
Mark Chan
Navdeep GILL
Patrick Hall
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
H2oAi Inc
Original Assignee
H2oAi Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/959,040 external-priority patent/US11922283B2/en
Priority claimed from US15/959,030 external-priority patent/US11386342B2/en
Application filed by H2oAi Inc filed Critical H2oAi Inc
Publication of EP3782079A1 publication Critical patent/EP3782079A1/fr
Publication of EP3782079A4 publication Critical patent/EP3782079A4/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/045—Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

  • Machine learning is a field of computer science that gives computers the ability to learn without being explicitly programmed.
  • a machine learning model can be trained to implement a complex function that makes one or more predictions based on a set of inputs.
  • the set of inputs is comprised of a plurality of entries. Each entry is associated with one or more features having corresponding feature values.
  • the machine learning model acts like a black box: it receives a set of inputs, the set of inputs are applied to the complex function, and one or more predictions are outputted.
  • Figure 1 is a block diagram illustrating an embodiment of a system for machine learning model interpretation.
  • Figure 2A is an example of a diagram illustrating an embodiment of input data.
  • Figure 2B is an example of a diagram illustrating an embodiment of input data that is ranked based on the prediction label.
  • Figure 3 is a diagram illustrating an embodiment of an output of a linear surrogate model.
  • Figure 4A is a flow chart illustrating an embodiment of a process for providing a linear surrogate model.
  • Figure 4B is a flow chart illustrating an embodiment of a process for providing a prediction.
  • Figure 5 is a diagram illustrating an embodiment of a non-linear surrogate model.
  • Figure 6 is a flow chart illustrating an embodiment of a process for providing a non linear surrogate model.
  • Figure 7 is a diagram illustrating an embodiment of a non-linear surrogate model.
  • Figure 8 is a flow chart illustrating an embodiment of a process for providing a surrogate non-linear model.
  • Figure 9 is a diagram illustrating an embodiment of a non-linear surrogate model.
  • Figure 10 is a flow chart illustrating an embodiment of a process for providing a non-linear model.
  • FIG. 11 is a diagram illustrating an embodiment of a dashboard.
  • Figure 12 is a flow chart illustrating an embodiment of a process for debugging machine learning models.
  • the invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.
  • these implementations, or any other form that the invention may take, may be referred to as techniques.
  • processor refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
  • a machine learning model interpretation technique is disclosed.
  • a machine learning model is configured to provide one or more predictions based on a set of inputs, however, it is unclear how the machine learning model arrived at its decision. Oftentimes the machine learning model is proprietary software of a company and users must receive a license to use the software.
  • the machine learning model may be limited in the type of information that is outputted to users.
  • the machine learning model may output a prediction, but may not provide one or more reasons why the machine learning model made the prediction. For example, the machine learning model may not output an identification of one or more input features that influenced the prediction of the machine learning model.
  • a machine learning model may be approximated a linear surrogate models and/or one or more non-linear surrogate models.
  • a surrogate model is a data mining and engineering technique in which a generally simpler model is used to explain another usually more complex model or phenomenon.
  • a surrogate model may reduce the number of computations and the time needed by a computer to output a prediction. The reduction in computations and time frees up computer resources, which allows the computer to perform other tasks and/or make other predictions.
  • a linear surrogate model may be a ⁇ G-LfME surrogate model.
  • a non-linear surrogate model may be a decision tree surrogate model, a feature importance surrogate model, and/or a partial dependence surrogate model.
  • a surrogate model may not only provide a prediction that is similar to the prediction made by the machine learning model, but also provide one or more reasons that describe why the surrogate model made its decision.
  • the combination of the linear surrogate model and the one or more non-linear surrogate models may provide confidence in the approximations.
  • the output of a linear surrogate model may closely match the output of the machine learning model, but the output of the linear surrogate model may be in conflict with the output of at least one of the non-linear surrogate models.
  • the output of a linear surrogate model may be in conflict with the output of the machine learning model, but the output of the one or more non-linear surrogate models closely matches the output of the machine learning model.
  • the combination of the linear surrogate model and the one or more non-linear surrogate models may be trusted to accurately explain the machine learning model of interest.
  • the combination of a linear surrogate model and the one or more non-linear surrogate models may reduce the number of computations and time needed by a computer to make a prediction when compared to the number of computations and time needed by a computer implementing the machine learning model to make the prediction.
  • the combination of a linear surrogate model and the one or more non-linear surrogate models provide transparency into the machine learning model.
  • the linear surrogate model and the one or more non-liner surrogate models allow the underlying machine learning model itself to be debugged.
  • FIG. 1 is a block diagram illustrating an embodiment of a system for machine learning model interpretation.
  • system 100 includes a complex model server 102, a network 105, a surrogate model server 112, and a client device 122.
  • Complex model server 102 includes a machine learning model 104, training data
  • Complex model server 102 may include one or more processors, one or more memories (e.g., random access memory), and one or more storage devices (e.g., read only memory).
  • Machine learning model 104 is configured to implement one or more machine learning algorithms (e.g., decision trees, naive Bayes classification, least squares regression, logistic regression, support vector machines, neural networks, deep learning, etc.).
  • Machine learning model 104 may be trained using training data, such as training data 116. Once trained, machine learning model 104 is configured to output a prediction label, such as model prediction data 107, based on an input entry that is comprised of one or more features and corresponding feature values.
  • Training Data 106 is comprised of a plurality of entries. Each entry is associated with one or more features having a corresponding feature value.
  • Model Prediction data 107 is comprised of predictions made by machine learning model 104.
  • Model prediction data 107 may include a probability of a particular outcome that the machine learning model has predicted.
  • Model prediction data 107 may include a prediction label (e.g., predicted value) for a particular prediction.
  • Actual Outcome data 108 is comprised of real world outcome data.
  • machine learning model 104 may be trained to predict the probability of a particular outcome given input data that is comprised of a plurality of entries.
  • Actual outcome data 108 includes the real world outcome for an entry associated with a plurality of features and corresponding feature values.
  • Network 105 may be a local area network, a wide area network, a wired network, a wireless network, the Internet, an intranet, or any other appropriate communication network.
  • Surrogate model server 112 includes a linear surrogate model 114, one or more surrogate non-linear models 115, training data 116, model prediction data 117, and actual outcome data 118.
  • the linear surrogate model 114, one or more surrogate non-linear models 115, training data 116, model prediction data 117, and actual outcome data 118 may be stored in memory and/or storage (not shown) of complex model server 112.
  • Surrogate model server 112 is configured to implement one or more surrogate models.
  • a surrogate model is a data mining and engineering technique in which a generally simpler model is used to explain another usually more complex model or phenomenon.
  • Surrogate model 112 may receive, from complex model server 102 via network 105, training data 106, model prediction data 107, and actual outcome data 108 and store as training data 116, model prediction data 117, and actual outcome data 118, respectively.
  • training data 116, model prediction data 117, and actual outcome data 118 surrogate model server 122 may train one or more surrogate models to make one or more predictions.
  • the one or more surrogate models are surrogates of machine learning model 104.
  • a surrogate model h may be trained, such that X, Y - > h, such that h(X) ⁇ g(X).
  • the surrogate model h may be a linear model or a non-linear model.
  • Linear surrogate model 114 may be a AT-I.IML surrogate model.
  • AT-I.IML local generalized linear model (GLM) surrogates are used to explain the predictions of complex response functions, and local regions are defined by K clusters or user-defined segments instead of simulated, perturbed observation samples.
  • LLM local generalized linear model
  • a local GLM h GLM k is trained.
  • the input data may be classified into a plurality of clusters using a clustering technique, such as k-means clustering.
  • K may be chosen such that predictions from all the local GLM models would maximize R 2 . This may be summarized mathematically as follows:
  • AT-I.IML may also train one global surrogate GLM h gi0i , ai on the entire input training dataset, such as training data 106 and global model predictions g(X), such as model prediction data 107.
  • GLM global surrogate
  • h si0t , ai is used as a linear surrogate instead of .
  • intercepts, coefficients, R 2 values, accuracy, and predictions from all the surrogate ⁇ -LfME models may be used to debug and increase transparency in g.
  • One or more reason codes and corresponding values may be generated from K-
  • a reason code corresponds to an input feature.
  • the reason code value may provide a feature’s approximate local, linear contribution to b (c®).
  • Reason codes are powerful tools for accountability and fairness because they provide an explanation for each b ( ®), enabling a user to understand the approximate magnitude and direction of an input feature’s local contribution for b (c®).
  • reason code values may be calculated by determining each coefficient-feature product.
  • Reason codes may also be written into automatically generated reason codes.
  • ST-LIME provides several scales of interpretability: (1) coefficients of the global
  • GLM surrogate provide information about global, average trends, (2) coefficients of in-segment GLM surrogates display average trends in local regions, and (3) when evaluated for specific in segment observations, ⁇ -LfME provides reason codes on a per-observation basis.
  • AT-I.IML may increase transparency by revealing input features and their linear trends.
  • ⁇ -LfME may enhance accountability by creating explanations for each observation in a data set.
  • ⁇ -LfME may bolster trust and fairness when the important features and their linear trends around specific records conform to domain knowledge and reasonable expectations.
  • the one or more surrogate non-linear models 115 may include a feature importance model, decision tree model, a partial dependence plot, and/or any other non-linear models.
  • a feature importance model measures the effect that a feature of the set of inputs has on the predictions of the model.
  • a feature may have a global feature importance and a local feature importance.
  • Global feature importance measures the overall impact of an input feature on the model predictions while taking nonlinearity and interactions into considerations.
  • Global feature importance values give an indication of the magnitude of a feature’s contribution to model predictions for all observations.
  • Local feature importance describes how the combination of the learned model rules or parameters and an individual observation’s attributes affect a model’s prediction for that observation while taking nonlinearity and interactions into effect.
  • the feature importance model may include a random forest surrogate model h RF consisting of B decision trees h tree b .
  • the random forest surrogate model is a global interpretability measure.
  • h RF may be expressed as:
  • 0 ft is the set of splitting rules for each tree h tree h .
  • the improvement in the split-criterion is the importance measure attributed to the splitting feature.
  • the importance feature is accumulated over all trees separately for each feature.
  • the aggregated feature importance values may be scaled between 0 and 1 , such that the most important feature has an importance value of 1.
  • Random forest feature importance increases transparency by reporting and ranking influential input features.
  • LOCO feature importance enhances accountability by creating explanations for each model prediction. Both global and local feature importance enhance trust and fairness when reported values conform to domain knowledge and reasonable expectations.
  • a decision tree model h tree may be generated to approximate the learned function g (e.g., machine learning model 104). h tree is used to increase the transparency of g by displaying an approximate flow chart of the decision making process of g. h tree also shows the likely important features and the most important interactions of g. h tree may be used for visualizing, validating, debugging g by comparing the displayed decision-process, important features, and important interactions to known standards, domain knowledge, and reasonable expectations.
  • a partial dependence plot may show how machine-learned response functions change based on the values of an input feature of interest, while taking nonlinearity into consideration and averaging out the effects of all other input features.
  • a partial dependence plot shows the partial dependence as a function of specific values of the feature subset X j . Partial dependence plots enable increased transparency in g and enable the ability to validate and debug g by comparing a feature’s average predictions across its domain to known standards and reasonable expectations.
  • the partial dependence plot includes an individual conditional expectation (ICE) plot.
  • ICE is a disaggregated partial dependence of the N responses e ⁇ 1, ... , N] (for a single feature X j ), instead of averaging the response across all observations of the training set.
  • the ICE plot may allow a prediction for an individual observation of data c/( ®) to determine whether the individual observation of data is outside one standard deviation from the average model behavior represented by partial dependence.
  • the ICE plot may also allow a prediction for an individual observation of data c/( ⁇ ) to determine whether the treatment of a specific observation is valid in comparison to average model behavior, known standards, domain knowledge, and/or reasonable expectations.
  • Training data 116 includes data that is used to train linear surrogate model 114 and/or one or more non-linear surrogate models 115. Training data 116 may include at least a portion of training data 106. Training Data 116 is comprised of a plurality of entries. Each entry is associated with one or more features having a corresponding value and associated actual outcomes.
  • Model Prediction data 117 is comprised of predictions made by machine learning model 104, predictions made by linear surrogate model 114, and predictions made by one or more non-linear surrogate models 115.
  • Model prediction data 117 may include a prediction label (e.g., probability of a particular outcome, predicted value, prediction value ⁇ offset value, etc.) that machine learning model 104 has predicted, a prediction label that linear surrogate model 114 has predicted, and a predication label that one or more non-linear surrogate models 115 has predicted.
  • a prediction label e.g., probability of a particular outcome, predicted value, prediction value ⁇ offset value, etc.
  • Actual Outcome data 118 is comprised of real world outcome data.
  • machine learning model 104, linear surrogate model 114, and one or more non-linear surrrogate models 115 may be trained to predict the probability of a particular outcome given a set of inputs.
  • Actual outcome data 118 includes the real world outcome given the set of inputs (e.g., did the particular outcome occur or not occur).
  • Client device 122 may be a computer, a laptop, a mobile device, a tablet, etc.
  • Client device 122 includes an application 124 associated with surrogate model server 112.
  • Application 124 is configured to display via graphical user interface 126, one or more graphs depicting the linear surrogate model 114 and at least one of the one or more non-linear surrogate models 115.
  • graphical user interface 126 is configured to receive a selection of a point (e.g., observation) shown in the linear surrogate model.
  • application 124 is configured to dynamically update the one or more non-linear surrogate models associated with the linear surrogate model and dynamically update a display of the one or more non-linear surrogate models.
  • Application 124 is also configured to provide an indication of the received selection to surrogate model server 112.
  • a linear surrogate model may be configured to provide one or more reason codes and corresponding reason code values to application 124.
  • a non-linear surrogate model may be configured to provide one or more important features for the selected point.
  • a non-linear surrogate model may be configured to highlight a decision tree path associated with the selected point.
  • Figure 2A is an example of a diagram illustrating an embodiment of input data.
  • Input data is comprised of training data, validation data, model prediction data, and actual outcome data.
  • input data 200 may be implemented by a system, such as complex model server 102 or surrogate model server 112.
  • input data 200 includes entries Ai, A ... A n .
  • Each entry is comprised of one or more features having a corresponding feature value.
  • entry Ai is comprised of features Fi, F ...Fn that have corresponding feature values of Xi
  • Y ...Z Entry A is comprised of features Fi
  • F ...Fn that have corresponding feature values of X
  • Y ...Z Entry An is comprised of features Fi, F ...Fn that have corresponding feature values of Xn, Yn... Zn.
  • a model such as machine learning model 104, linear surrogate model 114, or surrogate non-linear model(s) 115 may perform a prediction based on an entry, the features and corresponding feature values associated with the entry. For example, a model may output a prediction label Pi for Ai based on the features Fi, F ...F n and their corresponding feature values Xi, Y ...Z . A model may output a prediction of Pi, P ...P for each of the entries Ai, A ...A , respectively.
  • the prediction label may be a probability of a particular outcome, a predicted value, a predicted value plus an offset range, a predicted value plus a confidence level, etc.
  • Input data 200 may include actual outcome data, e.g., whether or not a particular outcome occurred, the actual value for an output variable, etc. A value of 1 may indicate that the particular outcome occurred. A value of 0 may indicate that the particular outcome did not occur.
  • a value of 1 indicates that the particular output did not occur and a value of 0 indicates that the particular outcome did occur.
  • a model such as machine learning model 104, linear surrogate model 114, or surrogate non-linear model(s) 115 may predict that a particular outcome is to occur (e.g., greater than or equal to a prediction threshold) and the particular outcome actually occurred (e.g., a value of 1).
  • a model such as machine learning model 104, linear surrogate model 114, or surrogate non-linear model(s) 115 may predict that a particular outcome is to occur (e.g., greater than or equal to a prediction threshold) and the particular outcome did not actually occurred (e.g., a value of 0).
  • a model such as machine learning model 104, linear surrogate model 114, or surrogate non-linear model(s) 115 may predict that a particular outcome is not to occur (e.g., less than a prediction threshold) and the particular outcome actually occurred (e.g., a value of 1).
  • a model such as machine learning model 104, linear surrogate model 114, or surrogate non-linear model(s) 115 may predict that a particular outcome is not to occur (e.g., less than a prediction threshold) and the particular outcome did not actually occur (e.g., a value of 0).
  • Figure 2B is an example of a diagram illustrating an embodiment of input data that is ranked based on the prediction label.
  • sorted training data 250 may be implemented by a system, such as complex model server 102 or surrogate model server 112.
  • input data 250 includes entries Ai, A ... A .
  • the entries for input data 250 are the same entries for input data 200, but ranked based on the prediction label.
  • the prediction label may be a probability of a particular outcome.
  • the entries are ranked from a lowest prediction label to the highest prediction label. In some embodiments, the entries are ranked from a highest prediction label to the lowest prediction label.
  • Figure 3 is a diagram illustrating an embodiment of an output of a linear surrogate model.
  • Linear model graph 300 may be implemented by a system, such as surrogate model server 112.
  • Linear model graph 300 may represent the output of a linear model, such as linear surrogate model 114.
  • Linear surrogate model 114 is a surrogate model of a more complex function, such as machine learning model 104.
  • Linear model graph 300 plots the prediction label associated with entries versus ranked predictions.
  • the y-axis of linear model graph 300 indicates a score made by a model, such as machine learning model 104 or linear surrogate model 114.
  • the x-axis of linear model graph 300 indicates a prediction ranking associated with a set of inputs.
  • the set of entries are ranked based on the prediction label and plotted sequentially.
  • Figure 2B depicts a set of entries that are ranked based on the corresponding prediction label.
  • the entries included in input data 250 would plotted in the following order: Ai, A ...and A .
  • Linear model graph 300 includes a line 301 that represents the prediction labels associated with a set of inputs that are determined by a machine learning model, such as machine learning model 104.
  • line 301 may be a plot of predictions Pi, P ...P of input data 250.
  • the prediction values associated with line 301 may be determined by a machine learning algorithm (e.g., decision trees, naive Bayes classification, least squares regression, logistic regression, support vector machines, neural networks, deep learning, etc.).
  • Linear model graph 300 includes a series of observations, for example, white dots
  • an observation is associated with a global surrogate model.
  • the observation may represent a prediction label of a global surrogate model for a particular entry.
  • an observation is associated with a local linear model.
  • the prediction label associated with each observation may be determined by a K-
  • Linear surrogate model 114 may be comprised of a plurality of local linear models.
  • the set of entries may be classified into one or more clusters using one or more techniques (e.g., k- means clustering).
  • Each cluster represents a subset of the entries that are similar to each other.
  • An entry may be associated with a cluster based on a distance between the entry and a cluster centroid. In the event the entry is less than or equal to a threshold distance away from a cluster centroid, an entry is associated with the cluster. In the event the entry is greater than a threshold distance away from a cluster centroid, an entry is associated with a different cluster.
  • a local linear model may be generated for each cluster.
  • the cluster local linear model may be trained using entries that are associated with a particular cluster.
  • each of the 11 clusters may have a corresponding local linear model.
  • Each local linear model is configured to make a prediction for the subset of entries that are included in a cluster.
  • a local linear model is configured to make a prediction based on the one or more features and corresponding feature values of an entry. For example, suppose white dot 302 is part of a first cluster and white dot 305 is part of a second cluster.
  • a first local linear model may be configured to generate a prediction for white dot 302 based on the one or more features and corresponding feature values of white dot 302 and a second local linear model may be configured to generate a prediction for white dot 305 based on the one or more features and corresponding feature values of white dot 305.
  • an entry is added to a cluster (e.g., production data) by determining a cluster centroid that is closest to the entry.
  • the entry and cluster centroids have a particular location in feature space.
  • an entry is comprised of a plurality of features and corresponding feature values.
  • the entry location in feature space may be represented as a vector, e.g., ⁇ Xi, Y ...Z ⁇ .
  • the closest cluster may be determined by computing a distance between the entry in the feature space and the cluster centroid in the feature space.
  • the closest cluster corresponds to a local linear model that has one or more associated model parameters.
  • a prediction label for the input may be determined by inputting the feature values associated with the feature to a local linear model that corresponds to the closest centroid cluster.
  • Linear model graph 300 includes a set of actual outcome data, for example, black dots 303, 304.
  • Black dots 303 indicate that the particular outcome actually occurred for entries having a set of features and corresponding feature values.
  • Black dots 304 indicate that the particular outcome did not occur for entries having a set of features and corresponding feature values.
  • Each of the observation points i.e., the white dots, has a corresponding black dot.
  • a global surrogate model correlates with the actual outcome data. In some embodiments, a global surrogate model does not correlate with the actual outcome data. In some embodiments, a local linear model prediction correlates with the actual outcome data. In some embodiments, a local linear model prediction does not correlate with the actual outcome data.
  • Each of the observation points may be selected.
  • one or more reason codes and corresponding reason code values may be displayed.
  • a reason code corresponds to a feature.
  • a reason code value corresponds to the amount that the feature contributed to the local model’s prediction label (e.g., weight) for that observation point (input point).
  • a linear surrogate model may determine the reason codes and corresponding reason code values for a particular observation point. The sum of the reason code values may be equal to the prediction label.
  • the top reason codes e.g., top 5 reason codes
  • the most influential features are displayed.
  • white dot 302 has a prediction label of approximately 0.3.
  • the top reason codes“FI,”“F18,”“F3,”“F50,”“F34,” and corresponding reason code values may be displayed.
  • selecting an observation point may cause all the reason codes and corresponding reason code values for the selected observation point to be displayed.
  • Figure 4A is a flow chart illustrating an embodiment of a process for providing a linear surrogate model.
  • process 400 may be implemented by a system, such as surrogate model server 112.
  • the data may include training data that was used to train the machine learning model.
  • the data may include prediction data of the machine learning model associated with an entry of the training data.
  • the data may include actual outcome data associated an entry with one or more features having a corresponding feature value, i.e., whether or not the particular outcome actually occurred.
  • the data associated with a machine learning model is classified into a plurality of clusters.
  • the data may be classified into the plurality of clusters using one or more techniques (e.g., k-means clustering).
  • Each cluster represents a subset of the entries that are similar to each other.
  • a cluster is comprised of a plurality of entries.
  • Each entry is comprised of one or more features having a corresponding feature value.
  • Each entry has a corresponding location, e.g., (Fi, F ...F n ) in a feature space.
  • a cluster is determined based on one or more entries that are within a threshold distance from a point (e.g., cluster centroid) in the feature space.
  • a model is created.
  • a global surrogate model is created based on the input data.
  • a separate linear model is created for each cluster.
  • Each linear model is configured to output a prediction label. For example, a linear model may determine a prediction Pi that indicates a probability of whether a particular outcome will occur given an entry Ai that is comprised of features Fi, F ... F n having corresponding feature values ofXi, Y ...Z
  • the entries are ranked based on a model prediction.
  • the entries are ranked based on a prediction made by a machine learning model, such as machine learning model 104.
  • the entries are ranked based on the prediction made by a linear surrogate model, such as linear surrogate model 114.
  • the entries are ranked from a lowest prediction label to the highest prediction label.
  • the entries are ranked from a highest prediction label to the lowest prediction label.
  • a linear model graph such as linear model graph 300
  • the linear model graph is provided from a surrogate model server to a client device via a network.
  • the client device may display the linear model graph via an application running on the client device.
  • a selection of an observation point included in the linear model graph is received.
  • a client device may receive via a GUI, a selection for a dot, such as white dot 302.
  • One or more non-linear model graphs may be updated based on the selected point.
  • the one or more reason codes include a set of features that predominately caused the entry to have the corresponding prediction label. For example, a series of reason codes may be provided to indicate why white dot 302 has a prediction label of 0.3. Each reason code has a corresponding reason code value that indicates a contribution to the prediction label. The cumulative contributions of the reason codes is equal to the prediction label.
  • FIG. 4B is a flow chart illustrating an embodiment of a process for providing a prediction.
  • Process 450 may be implemented by a system, such as surrogate model server 112.
  • Production data is received.
  • Production data is comprised of one or more entries. Each entry is associated with one or more features having corresponding feature values. The one or more entries of the production data do not include a corresponding prediction label.
  • a closest cluster is determined for each entry of the production data.
  • An entry of the production data is comprised of a plurality of feature values. The feature values correspond to a location in feature space.
  • a cluster centroid of a cluster has a corresponding location in the feature space.
  • a closest centroid is determined for each entry of the production data. The closest centroid may be determined by computing a distance between the location of an entry in feature space and a location of a cluster centroid in the feature space.
  • Each cluster has a corresponding linear surrogate model.
  • the one or more entries of production data are applied to a corresponding linear surrogate model.
  • a first entry of the production data may be applied to a first linear surrogate model that corresponds to a first cluster and a second entry of the production data may be applied to a second linear surrogate model that corresponds to a second cluster.
  • a prediction label and one or more reason codes are outputted.
  • Each linear surrogate model outputs a corresponding prediction label.
  • the prediction label may be a probability of a particular outcome, a predicted value, a prediction value ⁇ offset value, etc.
  • the reason codes provide an explanation as to why the prediction label has a certain output.
  • Figure 5 is a diagram illustrating an embodiment of a non-linear surrogate model.
  • Non-linear model graph 500 may be implemented by a system, such as surrogate model server 112.
  • Non-linear model graph 500 may represent the output of a non-linear surrogate model, such as one of the non-linear surrogate models 115.
  • a non-linear surrogate model 115 is a surrogate model of a more complex function, such as machine learning model 104.
  • Non-linear model graph 500 illustrates the feature importance of one or more features. Feature importance measures the effect that a feature has on the predictions of a model. Non-linear model graph 500 includes a global feature importance and a local feature importance for a particular feature. In some embodiments, the features are sorted in descending order from the globally most important feature to the globally least important feature.
  • the global feature importance measures the overall impact of the feature on the model predictions while taking nonlinearity and interactions into consideration.
  • a global feature importance value provides an indication of the magnitude of a feature’s contribution to model predictions for all observations.
  • the global importance value may indicate the importance of a feature for a global surrogate model, i.e., the importance of the feature for all entries.
  • the global feature importance value is equal to the number of times in a decision tree ensemble (e.g., global decision tree surrogate model) that a feature was selected to split a decision tree of the decision tree ensemble.
  • the global feature importance value is scaled to a number between 0 and 1 , such that the most important feature has an importance value of 1.
  • the global feature importance value is weighted based on a location of a feature in a decision tree. For example, a feature that is selected at the top of a decision tree for a split has a weight that is higher than another feature that is selected at the bottom of a decision tree for a split. In some embodiments, the weight is a value between 0 and 1. A weight of approximately 1 indicates that the feature was selected at or near the top of a decision tree. A weight of approximately 0 indicates that the feature was not selected for a branch of the decision tree or was selected at or near the bottom of a decision tree. In some embodiments, the weight is a value greater than 1.
  • Local feature importance describes how the combination of the learned model rules or parameters and an individual observation’s attributes affect a model’s prediction for that observation while taking nonlinearity and interactions into effect.
  • the local feature importance may indicate the importance of a feature associated with an entry (e.g., observation point) for a global surrogate model, i.e., the importance of the feature for this particular entry.
  • the local feature importance value may be determined by computing a LOCO value for a feature.
  • An entry is comprised of a plurality of features.
  • a first prediction is computed using the plurality of features and a second prediction is computed using the plurality of features less one of the plurality of features.
  • the second prediction is subtracted from the first prediction to determine the importance of the feature.
  • the LOCO value is computed for each feature of the plurality of features.
  • features“FI,”“F18,”“F3,”“F50,”“F34,” and“F8” are depicted as the most important features for a prediction.
  • the most important features are the most important features for a global surrogate model.
  • the most important features are the most important features for a selected observation point.
  • the global importance values and local importance values are shown for each feature.
  • the global importance values of 502a, 504a, 506a, 508a, 510a, and 512a are shown for features“FI,” “F18,”“F3,”“F50,”“F34,” and“F8,” respectively.
  • the local importance values of 502b, 504b, 506b, 508b, 510b, and 512b are shown for features“FI,”“F18,”“F3,”“F50,”“F34,” and“F8,” respectively.
  • the global importance value for a feature correlates with the local importance value for the feature. For example, the global importance value for a feature correlates with the local importance value for the feature in the event the difference between the two values is less than or equal to a threshold value. The global importance value for a feature does not correlate with the local importance value for the feature in the event the difference between the two values is greater than a threshold value. In the event the global importance value for a feature and the local importance value for the feature do not correlate, the entry with which the prediction is associated may be flagged. In some embodiments, the feature importance model is investigated to determine why the model outputted such values. In the event a threshold number of entries are flagged, the non-linear model may be determined to be inaccurate and adjusted.
  • the global importance value 504a for feature“FI 8” does not correlate with the local importance value 504b. This indicates that the non-linear model associated with non-linear model graph 500 may need to be adjusted or the feature importance model should be investigated.
  • the listed features may indicate that a single feature dominates the prediction label associated with a prediction (e.g., the feature importance value is greater than a dominance score).
  • feature FI may have an associated importance value of 0.98 (out of 1.00). This may indicate a data leak associated with the predication and indicate that the model may need to be adjusted or the feature importance model should be investigated. In response to such an indication, the model may be adjusted or investigated.
  • FIG. 6 is a flow chart illustrating an embodiment of a process for providing a non linear surrogate model.
  • process 600 may be implemented by a system, such as surrogate model server 112.
  • a global importance value of a feature is determined.
  • the global feature importance value may be equal to the number of times in a decision tree ensemble that the feature was selected to split a decision tree of the decision tree ensemble.
  • the global feature importance value is scaled to a number between 0 and 1 , such that the most important feature has an importance value of 1.
  • the global feature importance value is weighted based on a location of a feature in a decision tree. For example, a feature that is selected at the top of a decision tree for a split has a weight that is higher than another feature that is selected at the bottom of a decision tree for a split.
  • a local importance value of a feature is determined.
  • the local feature importance value may be determined by computing a LOCO value for a feature.
  • An entry is comprised of a plurality of features.
  • a first prediction is computed using the plurality of features and a second prediction is computed using the plurality of features less one of the plurality of features. The second prediction is subtracted from the first prediction to determine the importance of the feature.
  • the one or more most important features are ranked. In some embodiments, the one or more important features are ranked based on the global importance values. In other embodiments, the one or more important features are ranked based on the local importance values.
  • the top number e.g., top 5 of features or top percentage (top 10%) of features may be determined to be the one or more most important features.
  • a visualization of a comparison between the determined global importance value and the determined local importance for a plurality of features is provided.
  • the comparison is provided for the one or more most important features.
  • Figure 7 is a diagram illustrating an embodiment of a non-linear surrogate model.
  • Non-linear model graph 700 may be implemented by a system, such as surrogate model server 112.
  • Non-linear model graph 700 may represent the output of a non-linear surrogate model, such as one of the non-linear surrogate models 115.
  • a non-linear surrogate model 115 is a surrogate model of a more complex function, such as machine learning model 104.
  • Non-linear model graph 700 illustrates a decision tree surrogate model.
  • a complex decision tree ensemble model may be comprised of hundreds of trees with varying degrees of complexity (e.g., 1000s of levels).
  • the decision tree surrogate model is an approximation of the complex decision ensemble tree model (e.g., global decision tree surrogate model) and is comprised of a shallow decision tree, e.g., three levels.
  • Non-linear model graph 700 may indicate the most common decision path of a decision tree surrogate model.
  • a thickness of the most common decision path may have a greater thickness than other decision paths.
  • the path between“FI”,“F18,” and“F2” is thicker than other decision paths. This indicates that the path between“FI”,“F18,” and“F2” is the most common decision path for non-linear model graph 700.
  • Non-linear model graph 700 may indicate the least common decision path of a decision tree surrogate model.
  • a thinness of the least common decision path may have a thinner thickness than other decision paths. For example, the patch between“FI 8” and“F50” is thinner than other decision paths.
  • a width of a path of the decision tree surrogate model may indicate a frequency of which the path is used by the decision tree surrogate model.
  • Non-linear model graph 700 may include a prediction label associated with different paths associated with the decision tree surrogate model. For example, a prediction label of“0.136” is outputted for entries with features FI, F18, and F2.
  • non-linear model graph 700 may be updated to show the path of the observation through the decision tree surrogate model.
  • FIG 8 is a flow chart illustrating an embodiment of a process for providing a surrogate non-linear model.
  • process 800 may be implemented by a system, such as surrogate model server 112.
  • a decision tree surrogate model is generated.
  • a complex decision tree model may be comprised of hundreds of trees with varying degrees of complexity (e.g., 1000s of levels).
  • the decision tree surrogate model is an approximation of the complex decision tree model and is comprised of a shallow decision tree, e.g., three levels.
  • the linear surrogate model graph may plot the prediction labels of a linear surrogate model and a machine learning model with respect to ranked predictions.
  • An observation point is one of the predictions made by a linear surrogate model.
  • the decision tree surrogate model is updated based on the selected observation point.
  • the decision tree surrogate model may be updated to show the path of the selected observation point through the decision tree surrogate model.
  • Figure 9 is a diagram illustrating an embodiment of a non-linear surrogate model.
  • Non-linear model graph 900 may be implemented by a system, such as surrogate model server 112.
  • Non-linear model graph 900 may represent the output of a non-linear surrogate model, such as one of the non-linear surrogate models 115.
  • a non-linear surrogate model 115 is a surrogate model of a more complex function, such as machine learning model 104.
  • Non-linear model graph 900 illustrates a partial dependence plot.
  • a partial dependence plot determines the partial dependence of the prediction on a feature.
  • a partial dependence plot is configured to modify a feature value associated with a feature to be the same value for all entries and to determine the prediction label given the modified feature value.
  • an average prediction label is determined for different feature values.
  • non-linear graph 900 illustrates white dots that may have a value ranging from“-2” to“8.” The white dots depict the average prediction label for the inputs having the feature value.
  • Non-linear model graph 900 illustrates a range of prediction labels (e.g., one standard deviation) for all entries having the same feature value.
  • range 902 indicates that a model will usually output a prediction label between 0.1 and 0.4 when the feature value for a particular feature is“1.”
  • Non-linear model graph 900 illustrates a prediction label for an entry when a feature value is set to a particular value.
  • black dot 904 indicates that the prediction label is 0.2 when the feature value is set to“1” for the particular feature and particular entry.
  • Figure 10 is a flow chart illustrating an embodiment of a process for providing a non-linear model.
  • process 1000 may be implemented by a system, such as surrogate model server 122.
  • an indication to modify a feature value associated with a feature to be a particular value for all entries is received.
  • the feature is modified to be the particular value for all entries.
  • An entry is comprised of one or more features having a corresponding feature value.
  • the entry input data may indicate that the feature value for a particular feature varies for all entries.
  • the input data may be modified such that the feature value for a particular feature is the same for all entries.
  • the average prediction label for entries having the same feature value is determined.
  • the prediction label for all entries having the particular feature with the same feature value is computed and averaged.
  • the range of prediction labels (e.g., one standard deviation) for entries having the feature value is determined.
  • the prediction label for a single entry having the particular feature value is determined.
  • the single entry may correspond to a selected observation point in a linear surrogate model graph.
  • steps 1002-1008 is repeated for all possible values for a particular feature.
  • the feature depicted in Figure 9 has possible feature values of 2 to“8.”
  • Steps 1002-1008 may be repeated for when the feature value is“-2,”“-1,”...“8.”
  • FIG 11 is a chart illustrating an embodiment of a dashboard.
  • dashboard 1100 may be implemented by a system, such as surrogate model server 122.
  • Dashboard 1100 may be provided to a client system, such as client 122.
  • Dashboard 1100 may include a linear model graph and one or more non-linear model graphs, or graphs based on the original machine learning model.
  • dashboard 1100 includes a K-LIME linear model graph, a feature importance graph, a surrogate model decision tree, and a partial dependence graph.
  • a user selection of an observation such as white dot 1102 is received.
  • the feature importance graph, the surrogate model decision tree, and the partial dependence graph may be updated.
  • the feature importance graph may be updated to depict the most important features.
  • the most important features may be the most important features associated with a global surrogate model.
  • the most important features may be the most important features associated with the selected observation point.
  • the surrogate model decision tree may be updated to reflect a path in the surrogate decision tree that the observation took to arrive at the prediction label.
  • the partial dependence graph may be updated to depict how the prediction label for the observation point changes when the feature value of a particular feature is modified to be a particular value.
  • Figure 12 is a flow chart illustrating an embodiment of a process for debugging machine learning models.
  • process 1200 may be implemented by a system, such as surrogate model server 122.
  • the linear model graph may depict the predictions of a linear surrogate model.
  • the linear surrogate model graph may plot the prediction labels of a linear surrogate model and a machine learning model with respect to ranked predictions.
  • An observation point is one of the predictions made by a linear surrogate model.
  • one or more non-linear surrogate models are updated based on the selected point.
  • the feature importance graph may be updated to depict the most important features.
  • the surrogate model decision tree may be updated to reflect a path in the surrogate decision tree that the observation took to arrive at the prediction label.
  • the partial dependence graph may be updated to depict how the prediction label for the observation point changes when the feature value of a particular feature is modified to be a particular value.
  • an output of the linear surrogate model correlates with an output of the non-linear surrogate model.
  • an output of the linear surrogate model may indicate that feature“FI” is one of the top features that influenced the prediction label of the linear surrogate model while the output of a non-linear surrogate model indicates that feature “FI” is not one of the top features that influenced the prediction of the non-linear surrogate model.
  • process 1200 proceeds to 1210. In response to determining that the linear model does not agree with the linear model, process 1200 proceeds to 1212.
  • the linear surrogate model and/or at least one of the non-linear surrogate models are determined to be accurate.
  • the models are determined to be accurate because the explanations are deemed to be accurate. For example, determined feature importance, decision tree surrogate model outputs, and/or a partial dependence plot remaining stable over time or when training data is intentionally perturbed may be matched with human domain expertise to debug the models. In the event the explanations match with human domain expertise, then more confidence may be attached to the models.
  • These techniques may be used for visualizing, validating, and debugging the machine learning model by comparing the displayed decision-process, important features, and important interactions to known standards, domain knowledge, and reasonable expectations.
  • the linear and/or nonlinear model(s) are retrained.
  • the linear and/or non-linear surrogate models are retrained in the event a threshold number of entries are flagged. An entry may be flagged in the event a prediction label associated with a linear surrogate model does not correlate with a prediction label associated with a non-linear surrogate model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Algebra (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Selon la présente invention, des données d'entrée associées à un modèle d'apprentissage machine sont classées en une pluralité de groupes. Une pluralité de modèles de substitution linéaires sont générés. Un modèle de la pluralité de modèles de substitution linéaires correspond à un groupe de la pluralité de groupes. Un modèle de substitution linéaire est conçu pour fournir une prédiction correspondante sur la base de données d'entrée associées à un groupe correspondant. Des données de prédiction associées au modèle d'apprentissage machine et des données de prédiction associées à la pluralité de modèles de substitution linéaires sont fournis.
EP19788527.0A 2018-04-20 2019-04-08 Interprétation de modèle Pending EP3782079A4 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US15/959,040 US11922283B2 (en) 2018-04-20 2018-04-20 Model interpretation
US15/959,030 US11386342B2 (en) 2018-04-20 2018-04-20 Model interpretation
PCT/US2019/026331 WO2019204072A1 (fr) 2018-04-20 2019-04-08 Interprétation de modèle

Publications (2)

Publication Number Publication Date
EP3782079A1 true EP3782079A1 (fr) 2021-02-24
EP3782079A4 EP3782079A4 (fr) 2022-01-19

Family

ID=68239941

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19788527.0A Pending EP3782079A4 (fr) 2018-04-20 2019-04-08 Interprétation de modèle

Country Status (4)

Country Link
EP (1) EP3782079A4 (fr)
CN (1) CN112154459A (fr)
SG (1) SG11202009599SA (fr)
WO (1) WO2019204072A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111461862B (zh) * 2020-03-27 2023-06-30 支付宝(杭州)信息技术有限公司 为业务数据确定目标特征的方法及装置
CN113517035B (zh) * 2020-04-10 2024-02-02 中国石油天然气股份有限公司 表面活性剂的构效关系研究方法和装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7590642B2 (en) * 2002-05-10 2009-09-15 Oracle International Corp. Enhanced K-means clustering
US20140222349A1 (en) * 2013-01-16 2014-08-07 Assurerx Health, Inc. System and Methods for Pharmacogenomic Classification
US10452992B2 (en) * 2014-06-30 2019-10-22 Amazon Technologies, Inc. Interactive interfaces for machine learning model evaluations
WO2018017467A1 (fr) * 2016-07-18 2018-01-25 NantOmics, Inc. Systèmes, appareils et procédés d'apprentissage automatique distribué

Also Published As

Publication number Publication date
EP3782079A4 (fr) 2022-01-19
WO2019204072A1 (fr) 2019-10-24
CN112154459A (zh) 2020-12-29
SG11202009599SA (en) 2020-10-29

Similar Documents

Publication Publication Date Title
US11893467B2 (en) Model interpretation
US12118447B2 (en) Model interpretation
Kandanaarachchi et al. On normalization and algorithm selection for unsupervised outlier detection
US11875232B2 (en) Attributing reasons to predictive model scores
Mehdiyev et al. Evaluating forecasting methods by considering different accuracy measures
US10068186B2 (en) Model vector generation for machine learning algorithms
US6636862B2 (en) Method and system for the dynamic analysis of data
JP2021068436A (ja) 機械学習アルゴリズムの推論検証
Cano et al. International market selection using fuzzy weighing and Monte Carlo simulation
Nemet et al. Aggregation framework for TSK fuzzy and association rules: interpretability improvement on a traffic accidents case
Bosnić et al. Enhancing data stream predictions with reliability estimators and explanation
Amin et al. Cyber security and beyond: Detecting malware and concept drift in AI-based sensor data streams using statistical techniques
EP3782079A1 (fr) Interprétation de modèle
Pristyanto et al. The effect of feature selection on classification algorithms in credit approval
US11144938B2 (en) Method and system for predictive modeling of consumer profiles
Begum et al. Software Defects Identification: Results Using Machine Learning and Explainable Artificial Intelligence Techniques
Mohammadi et al. Hierarchical neural regression models for customer churn prediction
Ureña et al. On incomplete fuzzy and multiplicative preference relations in multi-person decision making
Perales-González et al. Negative correlation hidden layer for the extreme learning machine
Riesener et al. Identification of evaluation criteria for algorithms used within the context of product development
Darandale et al. Risk Assessment and Management using Machine Learning Approaches
Chakraborti et al. Evidential reasoning based digital twins for performance optimization of complex systems
Al Ali et al. Enhancing financial distress prediction through integrated Chinese Whisper clustering and federated learning
Sumalatha et al. Rough set based decision rule generation to find behavioural patterns of customers
Simarmata et al. Performance Evaluation of Classification Methods on Big Data: Decision Trees, Naive Bayes, K-Nearest Neighbors, and Support Vector Machines

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200929

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06N0003020000

Ipc: G06N0005040000

A4 Supplementary search report drawn up and despatched

Effective date: 20211222

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 7/00 20060101ALI20211216BHEP

Ipc: G06N 20/20 20190101ALI20211216BHEP

Ipc: G06N 5/00 20060101ALI20211216BHEP

Ipc: G06N 5/04 20060101AFI20211216BHEP

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230605

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20240717