CN112784986A - Feature interpretation method, device, equipment and medium for deep learning calculation result - Google Patents
Feature interpretation method, device, equipment and medium for deep learning calculation result Download PDFInfo
- Publication number
- CN112784986A CN112784986A CN202110178696.7A CN202110178696A CN112784986A CN 112784986 A CN112784986 A CN 112784986A CN 202110178696 A CN202110178696 A CN 202110178696A CN 112784986 A CN112784986 A CN 112784986A
- Authority
- CN
- China
- Prior art keywords
- feature
- sample
- value
- average
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The embodiment of the invention provides a method, a device, equipment and a medium for feature interpretation of deep learning calculation results, which can be used in the field of artificial intelligence or other fields, wherein the method comprises the following steps: inputting a preset sample data set into an original model to output a first predicted value, and calculating the average value of the first predicted value; inputting a preset data set into an explanation model for explaining an original model in SHAP to output a second predicted value, wherein the first predicted value is equal to the second predicted value; and calculating the average marginal contribution of each feature in the sample aiming at each sample in a preset sample data set, wherein the sum of the average marginal contributions of the features in the sample and the average value of the first predicted value is equal to the first predicted value corresponding to the sample, and the size of the average marginal contribution represents the importance degree of each feature in each sample to the calculation result of the original model. The scheme can quantitatively and visually explain the importance degree and the influence degree of the individual characteristics on the calculation result in the form of average marginal contribution.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a method, a device, equipment and a medium for feature interpretation of deep learning calculation results.
Background
From automation to augmented reality, Artificial Intelligence (AI) has silently changed the mode of business operations, and some companies have entered the era of automation. Artificial intelligence is gradually replacing human past work, such as fraud detection, resume review and loan applications, etc. Robots reduce the burden on employees to allow them to engage in higher levels of work. In the global CEO investigation published recently by the Yongdao of Puhua, people pay much attention to the risk and opportunity of artificial intelligence. Of these, 85% of CEOs believe that artificial intelligence will significantly change their business models in the next five years, and nearly two thirds of CEOs believe that the impact of artificial intelligence will far exceed the internet revolution. However, there is no clear representation of what the confidence in artificial intelligence is. In addition, more than three quarters of the CEOs consider artificial intelligence "socially beneficial", though more visitors (84%) consider that decisions made by artificial intelligence are reasonably reliable. The inability to interpret artificial intelligence decisions not only frustrates end users or customers, but also exposes organizations to operational, reputation, and financial risks. To trust an artificial intelligence system, it is necessary to let people "learn" their basic models in depth, study the data used by artificial intelligence, reveal the reasoning behind each decision, and provide consistent explanations to all stakeholders in time. Therefore, an evaluation mode for describing the artificial intelligence thinking process needs to be designed to realize the transparency of the artificial intelligence learning process and lay a foundation for realizing the interpretability of the artificial intelligence result, pursuing responsibility and following the moral regulation.
Disclosure of Invention
The embodiment of the invention provides a feature interpretation method of a deep learning calculation result, which aims to solve the technical problem that the interpretability of an artificial intelligence calculation result cannot be realized in the prior art. The method comprises the following steps:
inputting a preset sample data set into an original model, outputting a first predicted value by the original model, and calculating the average value of the first predicted value;
inputting the preset data set into an explanation model for explaining the original model in SHAP (cooperative game theory), wherein the explanation model outputs a second predicted value, and the first predicted value is equal to the second predicted value;
and calculating the average marginal contribution of each feature in the sample aiming at each sample in the preset sample data set, wherein the sum of the average marginal contributions of the features in the sample and the average value of the first predicted value is equal to the first predicted value corresponding to the sample, and the size of the average marginal contribution represents the importance degree of each feature in each sample to the calculation result output by the original model.
The embodiment of the invention also provides a device for interpreting the characteristics of the deep learning calculation result, which is used for solving the technical problem that the interpretability of the artificial intelligence calculation result cannot be realized in the prior art. The device includes:
the system comprises an original model calculation module, a first prediction value calculation module and a second prediction value calculation module, wherein the original model calculation module is used for inputting a preset sample data set into an original model, the original model outputs the first prediction value, and the average value of the first prediction value is calculated;
the interpretation model calculation module is used for inputting the preset data set into a SHAP (secure short-range access point) to interpret an interpretation model of the original model, and the interpretation model outputs a second predicted value, wherein the first predicted value is equal to the second predicted value;
and the characteristic interpretation module is used for calculating the average marginal contribution of each characteristic in each sample in the preset sample data set, wherein the sum of the average marginal contributions of the characteristics in the sample and the average value of the first predicted value is equal to the first predicted value corresponding to the sample, and the size of the average marginal contribution represents the importance degree of each characteristic in each sample to the calculation result output by the original model.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the above arbitrary feature interpretation method of the deep learning calculation result when executing the computer program so as to solve the technical problem that the interpretability of the artificial intelligence calculation result cannot be realized in the prior art.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program for executing the feature interpretation method of the arbitrary artificial intelligence calculation result, so as to solve the technical problem that the interpretability of the calculation result cannot be realized in the deep learning in the prior art.
In the embodiment of the invention, the average marginal contribution of each feature in a sample is calculated, and then based on the game theory principle, the sum of the average marginal contribution of each feature in the sample and the average value of the first predicted values of each sample in a preset sample data set is equal to the first predicted value corresponding to the sample, wherein the first predicted value corresponding to the sample is equal to the second predicted value corresponding to the sample, namely, the average marginal contribution of each feature meeting uniqueness can be calculated; in addition, the average marginal contribution of each feature represents the importance degree of each feature in each sample to the calculation result output by the original model, so that the importance degree and the influence degree of the individual feature on the calculation result can be quantitatively and visually explained in the form of the average marginal contribution, and a foundation is laid for the requirements of realization of interpretability, traceability, adherence to moral regulations and the like of the artificial intelligence calculation result.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 is a flowchart of a method for feature interpretation of deep learning computation results according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating an individual characteristic explanation by taking a customer marketing case as an example according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating cross-interpretation of two features of balance and duration by taking a customer marketing case as an example according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating cross-interpretation of two features of balance and age by taking a customer marketing case as an example according to an embodiment of the present invention;
FIG. 5 is a schematic diagram illustrating cross interpretation of two features, age and education, by taking a customer marketing case as an example, according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a positive-negative relationship between a Logit value and a predicted value of a feature provided in an embodiment of the present invention;
fig. 7 is a schematic diagram illustrating how individual feature balance affects the calculation result by taking a customer marketing case as an example according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating how individual feature duration affects the calculation result by taking a customer marketing case as an example according to an embodiment of the present invention;
fig. 9 is a schematic diagram illustrating how the individual feature age affects the calculation result by taking a customer marketing case as an example according to an embodiment of the present invention;
FIG. 10 is a schematic diagram illustrating how an individual characteristic day affects a calculation result by taking a customer marketing case as an example according to an embodiment of the present invention;
FIG. 11 is a schematic diagram illustrating how individual characteristics month influence the calculation result by taking a customer marketing case as an example, according to an embodiment of the present invention;
fig. 12 is a schematic diagram illustrating how the individual characteristic jobs affects the calculation result by taking a customer marketing case as an example, according to an embodiment of the present invention;
FIG. 13 is a schematic diagram illustrating how individual characteristic education affects the calculation result by taking a customer marketing case as an example according to an embodiment of the present invention;
FIG. 14 is a diagram illustrating individual feature interpretation taking prediction of a certain customer sample as an example according to an embodiment of the present invention;
FIG. 15 is a diagram illustrating how individual characteristics affect the final prediction result by taking the prediction of a certain customer sample as an example according to an embodiment of the present invention;
FIG. 16 is a block diagram of a computer device according to an embodiment of the present invention;
fig. 17 is a block diagram of a feature interpretation apparatus for a deep learning calculation result according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
In an embodiment of the present invention, a method for feature interpretation of a deep learning calculation result is provided, as shown in fig. 1, the method including:
step 102: inputting a preset sample data set into an original model, outputting a first predicted value by the original model, and calculating the average value of the first predicted value;
step 104: inputting the preset data set into an explanation model for explaining the original model in SHAP, wherein the explanation model outputs a second predicted value, and the first predicted value is equal to the second predicted value;
step 106: and calculating the average marginal contribution of each feature in each sample in the preset sample data set, wherein the sum of the average marginal contributions of the features in the sample and the average value of the first predicted value is equal to the first predicted value corresponding to the sample, and the average marginal contribution represents the importance degree of each feature in each sample to the calculation result output by the original model (for example, the larger the average marginal contribution is, the larger the importance degree of the feature to the calculation result output by the original model is, and the smaller the average marginal contribution is, the smaller the importance degree of the feature to the calculation result output by the original model is).
As can be seen from the process shown in fig. 1, in the embodiment of the present invention, calculating the average marginal contribution of each feature in a sample is provided, and then based on the game theory principle, the sum of the average marginal contribution of each feature in the sample and the average of the first predicted values of each sample in a preset sample data set is equal to the first predicted value corresponding to the sample, where the first predicted value corresponding to the sample is equal to the second predicted value corresponding to the sample, that is, the average marginal contribution of each feature meeting uniqueness can be calculated; in addition, the average marginal contribution of each feature represents the importance degree of each feature in each sample to the calculation result output by the original model, so that the importance degree and the influence degree of the individual feature on the calculation result can be quantitatively and visually explained in the form of the average marginal contribution, and a foundation is laid for the requirements of realization of interpretability, traceability, adherence to moral regulations and the like of the artificial intelligence calculation result.
In specific implementation, the feature interpretation method for the deep learning calculation result provides that the average marginal contribution of each feature in each sample is calculated, the importance and the contribution degree of each feature to the calculation result are measured by the average marginal contribution, that is, a game theory principle is applied, each feature variable in the deep learning original model is used as one player, the sample data set is used for training the original model to obtain a prediction result, the calculation result can be regarded as the profit of a plurality of players for completing one project cooperatively, and the measurement of the importance of each feature to the original model is realized by evaluating the contribution degree of each player to the calculation result. For this reason, the principle of simple Additive exhibition (referred to as SHAP for short) is introduced to explain the complex machine learning model, and the core of SHAP is to calculate the Shapley Value of each feature variable in the SHAPLY Additive exhibition.
Specifically, when performing the post-hoc explanation of the SHAP of the machine learning model, explicit labeling is required. The method comprises the steps of knowing a sample data set (the data set is provided with M characteristic variables, a samples), an original model f and all predicted values f (x) of the original model f on the sample data set. g is the interpretation model used in SHAP to interpret the original model f. The process of calculating the shape value for the machine learning model is as follows: firstly, predicting the sample data set by using an original model f to obtain an original modelf average value of the predicted values (i.e., the first predicted values) output; a single sample is denoted x ═ x (x)1,x2,…,xM) F (x) is the predicted value of the sample data set under the original model, g (x) is the predicted value of the post-explanation model, and the predicted value satisfies the following conditions:
wherein the content of the first and second substances,
shape value (i.e. average marginal contribution) representing the ith characteristic variable is a value to be calculated by the core in the SHAP, at this time, the average marginal contribution needs to satisfy uniqueness, and the corresponding interpretation model g is also unique, that is, the interpretation model g needs to satisfy the following properties:
properties 1: local fidelity (local accuracy)
I.e. the predicted values obtained by the two models are equal. When a single sample x is input into the interpretation model g, the resulting prediction value
Equal to the predicted value f (x) obtained from the original model.
Properties 2: deletion (missingness)
If a missing value exists in a single sample, namely no value is taken under a certain characteristic variable, no influence is caused on the interpretation model g, and the shape value is 0.
Properties 3: continuity (consistency)
When the original model f is changed from a random forest to XGboost, if the contribution of a characteristic variable to the model prediction value is increased, the shape value of the original model is increased.
In specific implementation, the shape value of a feature is an average marginal contribution of the feature in all feature sequences, and the marginal contribution of the feature when the feature is added to the model is calculated first, and then different marginal contributions of the feature in all feature sequences are averaged, that is, the average value is the SHAP baseline value of the feature. In order to achieve quantitative and visual measurement of the importance and influence of the individual features on the deep learning calculation result, in the present embodiment, a specific way of calculating the average marginal contribution of the individual features is proposed, for example, the average marginal contribution of each feature in the sample is calculated by the following formula:
wherein the content of the first and second substances,
representing the average marginal contribution of the ith feature; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features of the sample; f. ofxA function representing the original model; f. ofx(S∪{i})-fx(S) represents the amount of increase in the contribution margin when the ith feature is added to the ranked subset S.
In specific implementation, the global interpretability of the individual features can be realized by calculating the average marginal contribution of the individual features, the overall importance of the individual features to the global features is explained, and then what features have the greatest influence on model prediction is explained, for example, a case of accurate customer marketing is taken as a calculation case of individual feature interpretation, the average marginal contribution result of each feature calculated by the feature interpretation method of the deep learning calculation result is shown in fig. 2, and as can be seen from fig. 2, the six features having the greatest influence on the model calculation result are respectively: balance (annual account average balance), duration (communication duration in the marketing process, min), age (age), day (marketing time, number of days), month (marketing time, months), job (job type).
In specific implementation, in order to explain the influence and importance of multiple feature intersections on the deep learning calculation result, in this embodiment, it is further proposed to calculate an average marginal contribution of the multiple feature intersections in each sample, for example, calculate the average marginal contribution of the multiple feature intersections in the sample by the following formula:
wherein the content of the first and second substances,
representing the average marginal contribution of the t feature intersections; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features; n is a feature set composed of t features in N, fxA function representing the original model;
indicating the amount of increase in the contribution margin when t features in n are added to the ranked subset S.
Specifically, for example, the value of t is 2, and the average marginal contribution of two feature intersections is calculated as:
i≠j
wherein the content of the first and second substances,
indicating the amount of increase in the contribution margin when both features i, j are added to the ranked subset S.
In specific implementation, the global interpretability of the multiple feature intersections can be realized by calculating the average marginal contribution of the multiple feature intersections, the overall importance of the multiple feature intersections to the global features is explained, and how the cross effect of the multiple features influences the prediction of the model is further explained. For example, taking the case of accurate customer marketing shown in fig. 2 as a calculation case of cross interpretation of a plurality of features, we can calculate the average marginal contribution of intersection between two features in three features, namely, the balance, duration, and age, with the greatest feature importance by using the feature interpretation method of the deep learning calculation result.
Specifically, the average marginal contribution of intersection of the balance and the duration is shown in fig. 3, an X axis in fig. 3 is a value of the feature balance, a Y axis is an average marginal contribution value of intersection corresponding to the feature balance and the duration of one customer sample, a right side in fig. 3 is a value of the feature duration, and the value of the duration changes from small to large, so that the relationship between the balance and the duration is described, most of the values of the duration (i.e., the contact time with the customer) in the batch of samples do not exceed 400s, for the same feature balance value (i.e., customers with the same deposit), the influence of the greater the value of the duration on the model prediction is more obvious, the average marginal contribution of the increase of the balance value and the duration value is also increased, that is, the average marginal contribution of the balance value and the duration value is positive, that is, that increasing the contact time in the marketing process can increase the effect of the model prediction on the positive sample.
Specifically, the average marginal contribution of intersection of balance and age is shown in fig. 4, the X axis is a value of the feature balance, the Y axis is an average marginal contribution value of intersection corresponding to the feature balance and age of one customer sample, the right side of fig. 4 is a value of the feature age, and the value of age varies from small to large. That is, for customers with the same credit, the effect of middle-aged customers on model prediction as a positive sample is more significant.
Specifically, the average marginal contribution of the intersection of the education and the age is shown in fig. 5, the X axis is the value of the feature age, the Y axis is the average marginal contribution value of the intersection corresponding to the feature education and the age of one customer sample, the right side of fig. 5 is the feature education, the education corresponds to primary school education (primary), secondary school education (secondary school), and university education (tertiary school), it can be seen that, excluding the influence of all other features, the relationship between the education and the age is described, the age values of most customers in this batch of samples fall within the age range of 20-60, the education is above the secondary school, the age value is about 20-25 years old, the interaction between the education and the secondary school education has a larger value, the interaction between the two has a larger effect on the model prediction as a positive sample, and the two features interact to make the customers tend to purchase the product; the age value is 30-60 years old, the cross average marginal contribution value of the education is not large, and the interaction of the age value and the education has little influence on model prediction.
In specific implementation, in order to unify the value range deviation of each feature information in the algorithm process, the significance of the feature needs to be described by introducing a Logit value. For example, to further explain whether an individual feature has a positive influence or a negative influence on a deep learning result, in this embodiment, it is proposed to calculate a Logit value (i.e., an evaluation model value) of each feature according to an average marginal contribution of each feature in each sample, where a trend of the Logit value is proportional to a positive or negative influence of each feature on a calculation result output by the original model, that is, an increase in the Logit value indicates that an influence of a feature on the calculation result output by the original model is a positive influence, and a decrease in the Logit value indicates that an influence of a feature on the calculation result output by the original model is a negative influence.
Specifically, in statistics, Probability (Proavailability) and Odds are used to describe the Probability of something happening. Probability describes the ratio of the number of occurrences of an event a to the number of occurrences of all the results. The probability is a real number between 0 and 1. Unlike probabilities, Logit has an important property of having no upper and lower bounds — which brings great convenience to modeling. The relationship between the probability and the Logit value is:
as shown in fig. 6, when the probability p > 0.5, the model prediction is positive, and the Logit value is positive; and when the probability p is less than or equal to 0.5, the model prediction is negative, and the Logit value is negative.
It can be seen that the Logit value increases with the increase of the probability P, that is, the interpretation of the classification probability is performed by using the Logit value instead of the probability, and then the positive and negative influences of each feature on the calculation result output by the original model are interpreted according to the variation trend of the Logit value of each feature. Specifically, P in the above formula may be replaced by the average marginal contribution of each feature, so as to calculate the Logit value of each feature.
Specifically, the global interpretability of positive and negative influences of each feature on the calculation result can be realized by calculating the Logit value of each feature, and then how each feature influences the prediction of the model is explained by considering a large number of samples as a whole. For example, the case of accurate customer marketing shown in fig. 2 is used as a calculation case for cross interpretation of a plurality of features, and by the feature interpretation method of the deep learning calculation result, the log values of six features with the maximum feature importance and the relatively insignificant feature education can be calculated respectively.
For example, the Logit value of balance is shown in fig. 7, the abscissa represents the value of the characteristic balance, and the ordinate represents the Logit average value below the same balance value. The portion with a Logit average greater than-7.534 represents a higher probability of predicting the model as a positive sample (and thus pushing to a prediction of 1, positive output), and the portion with a Logit average less than-7.534 represents a lower probability of predicting the model as a positive sample (and thus pushing to a prediction of 0, negative output). As can be seen from fig. 7: the origin of coordinates is-7.534, i.e., the basic logit value for the positive category is-7.534; when the balance deposit number is negative, reducing the logic value, namely reducing the probability that the model is predicted to be a positive type, namely the balance negatively affects the model prediction, namely a characteristic balance trend client does not purchase the product; and when the balance deposit is positive, increasing the local value to increase the probability of predicting as a positive category, namely the balance positively influences the model prediction, namely a characteristic balance trend customer purchases the product. When the balance deposit number is around 32000 and 58000, the value of logit can be increased to the maximum extent, namely, the customer is predicted to be most driven to purchase the product.
Specifically, the Logit value of the duration is as shown in fig. 8, and the abscissa in fig. 8 represents the value of the characteristic duration, and the ordinate represents the Logit average value below the same duration value. The portion of the Logit average greater than-7.534 represents a higher probability of predicting the model as a positive sample (positive output) and the portion of the Logit average less than-7.534 represents a lower probability of predicting the model as a positive sample (negative output). As can be seen from fig. 8: when the communication duration in the duration marketing process is less than or equal to 200min, driving to reduce the logit value, and driving the model prediction to be a negative type, namely, the duration plays a negative influence on the model prediction; when the communication duration in the duration marketing process is more than 200min, the logic value is driven to be increased, the model prediction is driven to be in a positive type, namely, the duration plays a positive influence on the model prediction. It is generally believed that increasing the marketing communication time after the communication duration exceeds 1200min during the marketing process does not help the model to predict as positive category.
Specifically, the Logit value of an age is as shown in fig. 9, and in fig. 9, the abscissa represents the value of a characteristic age, and the ordinate represents the Logit average value below the same age value. The portion of the Logit average greater than-7.534 represents a higher probability of predicting the model as a positive sample (positive output) and the portion of the Logit average less than-7.534 represents a lower probability of predicting the model as a positive sample (negative output). As can be seen from fig. 9: when the age falls in the range of 35-60 years, the logic value is driven to be reduced, the model prediction is driven to be in a negative category, namely, the age has negative influence on the model prediction; before age 35 and after age 60, drives up the logit value, driving the model prediction to positive class, i.e. age has a positive effect on the model prediction. In the range of 20-30 years, the effect of driving increase of logit value is smaller and smaller with increasing age, almost no effect is achieved in the vicinity of 30 years, and the effect is maximum in the vicinity of 20 years. The effect of driving an increase in logit values is greater in the range of 60-80 years, with the effect being greatest near 62, 71, 78.
Specifically, the Logit value of day is shown in fig. 10, where the abscissa in fig. 10 represents the value of characteristic day and the ordinate represents the Logit average value below the same day value. The portion of the Logit average greater than-7.445 represents a higher probability of predicting the model as a positive sample (positive output) and the portion of the Logit average less than-7.445 represents a lower probability of predicting the model as a positive sample (negative output). As can be seen from fig. 10: before the fifth, about 10-15 and 25 of each month, marketing is carried out, a logit value is driven to be increased, and model prediction is driven to be a positive type, namely day plays a positive influence on the model prediction; and marketing is carried out at other times, the trend reduces the value of the logic, the trend model is predicted to be in a negative category, and day has a negative influence on the model prediction. Marketing before five months each, the effect of driving the increase in logit values is less and less, but marketing at the beginning of the month, especially marketing nos. 2-4, in terms of the entire month, day has the greatest positive effect on the predictive outcomes.
Specifically, the Logit value of month is shown in fig. 11, where the abscissa in fig. 11 represents the value of characteristic month, and the ordinate represents the Logit average value below the same month value. The portion where the Logit average is greater than-7.491 represents the probability of predicting the model as a positive sample being pushed up (positive output), and the portion where the Logit average is less than-7.491 represents the probability of predicting the model as a positive sample being pushed down (negative output). Month is the category data, and after pre-encoding the category data, the encoding corresponding to Month is shown in the following table 1, as can be seen from FIG. 11: the Logit values corresponding to the numbers 4, 5 and 6 are unchanged, and the represented january, july and june have almost no influence on model prediction; the Logit values corresponding to the numbers 8 and 9 are reduced, and the represented May and November have negative influence on model prediction; the Logit values corresponding to other months are increased, namely the influence on model prediction is positive; the Logit value corresponding to numbers 7 and 11 is the largest, and the marketing in march and september represented has the largest influence on the positive category predicted by the model, namely, the most possible customers buy the product.
TABLE 1
| Apr：：0 | Feb：3 | Jun：6 | Nov：9 |
| Aug：：1 | Jan：4 | Mar：7 | Oct：10 |
| Dec：2 | Jul：5 | May：8 | Sep：11 |
Specifically, the Logit value of a joba is as shown in fig. 12, where the abscissa in fig. 12 represents the value of a characteristic joba, and the ordinate represents the Logit average value below the same joba value. The portion of the Logit average greater than-7.445 represents a higher probability of predicting the model as a positive sample (positive output) and the portion of the Logit average less than-7.445 represents a lower probability of predicting the model as a positive sample (negative output). jobs are category data, and after precoding the category data, the codes corresponding to jobs are shown in table 2 below, as can be seen from fig. 12: the Logit values corresponding to numbers 4, 5, 6 and 7 are unchanged, and management, retered, self-organization and unemployed represented by the numbers have almost no influence on model prediction; the Logit values corresponding to the numbers 1, 2, 3 and 10 are reduced, and the represented blue-collar, entrepreneur, housmemid and unemployed have negative influence on model prediction; logit values corresponding to other professions increase, i.e. the impact on model prediction is a positive impact; the Logit values of numbers 0 and 9 are the largest, representing admin, Technician occupational type, with the greatest impact on the positive category predicted by the model, i.e., most likely to enable the customer to purchase the product.
TABLE 2
Specifically, the Logit value of reduction is shown in fig. 13, where the abscissa in fig. 13 represents the value of the characteristic reduction, and the ordinate represents the Logit average value below the same reduction value. The portion of the Logit average greater than-7.445 represents a higher probability of predicting the model as a positive sample (positive output) and the portion of the Logit average less than-7.445 represents a lower probability of predicting the model as a positive sample (negative output). The reduction is the category data, and after the category data is pre-coded, the coding corresponding to the reduction is shown in the following table 3, as can be seen from fig. 13: the Logit corresponding to the numbers 0 and 1 is reduced, which represents that primary education and secondary education have negative influence on model prediction; the Logit increase corresponding to number 2 represents that tertiary (university education) is a positive impact on model prediction.
TABLE 3
| primary：0 | secondary：1 | tertiary：2 |
In specific implementation, regarding the prediction of a certain customer sample, how each feature affects the final prediction result can be explained through the feature interpretation method of the deep learning calculation result. The following analysis randomly extracts one sample data from a test set for analysis, and fig. 14 shows one sample record, wherein feature represents each feature included in the sample record, feature _ value represents the value of each feature, and shape _ value represents the average boundary contribution of each feature, as shown in fig. 15By passing
The predicted value of the sample is calculated to be-10.26, and meanwhile, fig. 15 also shows the size of the average boundary contribution of each feature in the sample in a section mode, and further highlights the influence of each feature on the predicted value when the feature participates in the predicted value operation through the size of the average boundary contribution, for example, the longer the section is, the larger the average boundary contribution is, and the larger the influence on the predicted value is.
In this embodiment, a computer device is provided, as shown in fig. 16, which includes a memory 1602, a processor 1604, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the processor implements any of the above-mentioned feature interpretation methods for the deep learning calculation results.
In particular, the computer device may be a computer terminal, a server or a similar computing device.
In the present embodiment, there is provided a computer-readable storage medium storing a computer program for executing the feature interpretation method of the result of the deep learning calculation as described above.
In particular, computer-readable storage media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable storage medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
Based on the same inventive concept, the embodiment of the present invention further provides a feature interpretation apparatus for deep learning calculation results, as described in the following embodiments. Because the principle of the feature interpretation device for the deep learning calculation result for solving the problem is similar to the feature interpretation method for the deep learning calculation result, the implementation of the feature interpretation device for the deep learning calculation result can refer to the implementation of the feature interpretation method for the deep learning calculation result, and repeated details are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 17 is a block diagram showing a configuration of a feature interpretation apparatus for a deep learning calculation result according to an embodiment of the present invention, and as shown in fig. 17, the apparatus further includes:
an original model calculation module 1702, configured to input a preset sample data set into an original model, where the original model outputs a first predicted value, and calculates an average value of the first predicted value;
an interpretation model calculation module 1704, configured to input the preset data set into a SHAP to interpret an interpretation model of the original model, where the interpretation model outputs a second predicted value, and the first predicted value is equal to the second predicted value;
a feature interpretation module 1706, configured to calculate, for each sample in the preset sample data set, an average marginal contribution of each feature in the sample, where a sum of the average marginal contributions of each feature in the sample and an average of the first predicted values is equal to the first predicted value corresponding to the sample, where a size of the average marginal contribution represents a degree of importance of each feature in each sample to a calculation result output by the original model.
In one embodiment, the feature interpretation module calculates the average marginal contribution of each feature in the sample by:
wherein the content of the first and second substances,
representing the average marginal contribution of the ith feature; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features of the sample; f. ofxA function representing the original model; f. ofx(S∪{i})-fx(S) represents the amount of increase in the contribution margin when the ith feature is added to the ranked subset S.
In one embodiment, the feature interpretation module is further configured to calculate an average marginal contribution of the plurality of feature intersections in the sample by:
wherein the content of the first and second substances,
representing the average marginal contribution of the t feature intersections; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features; n is a feature set composed of t features in N, fxA function representing the original model; vn(S) represents the amount of increase in the contribution margin when t features in n are added to the ranked subset S.
In one embodiment, the feature interpretation module is further configured to calculate a Logit value of each feature according to the average marginal contribution of each feature in each sample, wherein a variation trend of the Logit value is proportional to a positive or negative influence of each feature on a calculation result output by the original model.
The embodiment of the invention realizes the following technical effects: the average marginal contribution of each feature in the sample is calculated, and then based on the game theory principle, the sum of the average marginal contribution of each feature in the sample and the average value of the first predicted value of each sample in the preset sample data set is equal to the first predicted value corresponding to the sample, wherein the first predicted value corresponding to the sample is equal to the second predicted value corresponding to the sample, namely, the average marginal contribution of each feature meeting uniqueness can be calculated; in addition, the average marginal contribution of each feature represents the importance degree of each feature in each sample to the calculation result output by the original model, so that the importance degree and the influence degree of the individual feature on the calculation result can be quantitatively and visually explained in the form of the average marginal contribution, and a foundation is laid for the requirements of realization of interpretability, traceability, adherence to moral regulations and the like of the artificial intelligence calculation result.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for feature interpretation of deep learning calculation results, comprising:
inputting a preset sample data set into an original model, outputting a first predicted value by the original model, and calculating the average value of the first predicted value;
inputting the preset data set into an interpretation model for interpreting the original model in a cooperative game theory, wherein the interpretation model outputs a second predicted value, and the first predicted value is equal to the second predicted value;
and calculating the average marginal contribution of each feature in the sample aiming at each sample in the preset sample data set, wherein the sum of the average marginal contributions of the features in the sample and the average value of the first predicted value is equal to the first predicted value corresponding to the sample, and the size of the average marginal contribution represents the importance degree of each feature in each sample to the calculation result output by the original model.
2. The method of claim 1, wherein computing the average contribution to margin of each feature in the sample comprises:
the average marginal contribution of each feature in the sample is calculated by the following formula:
wherein the content of the first and second substances,
representing the average marginal contribution of the ith feature; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features of the sample; f. ofxA function representing the original model; f. ofx(S∪{i})-fx(S) represents the amount of increase in the contribution margin when the ith feature is added to the ranked subset S.
3. The feature interpretation method of the deep learning calculation result according to claim 1 or 2, further comprising:
calculating the average marginal contribution of the plurality of feature intersections in the sample by the following formula:
wherein the content of the first and second substances,
representing the average marginal contribution of the t feature intersections; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features; n is a feature set composed of t features in N, fxA function representing the original model;
indicating the amount of increase in the contribution margin when t features in n are added to the ranked subset S.
4. The feature interpretation method of the deep learning calculation result according to claim 1 or 2, further comprising:
and calculating the evaluation model value of each feature according to the average marginal contribution of each feature in each sample, wherein the change trend of the evaluation model value is in direct proportion to the positive and negative of the influence of each feature on the calculation result output by the original model.
5. A feature interpretation apparatus for a deep learning calculation result, comprising:
the system comprises an original model calculation module, a first prediction value calculation module and a second prediction value calculation module, wherein the original model calculation module is used for inputting a preset sample data set into an original model, the original model outputs the first prediction value, and the average value of the first prediction value is calculated;
the interpretation model calculation module is used for inputting the preset data set into an interpretation model of the original model in a cooperative game theory, and the interpretation model outputs a second predicted value, wherein the first predicted value is equal to the second predicted value;
and the characteristic interpretation module is used for calculating the average marginal contribution of each characteristic in each sample in the preset sample data set, wherein the sum of the average marginal contributions of the characteristics in the sample and the average value of the first predicted value is equal to the first predicted value corresponding to the sample, and the size of the average marginal contribution represents the importance degree of each characteristic in each sample to the calculation result output by the original model.
6. The apparatus for feature interpretation of deep learning calculation results according to claim 5, wherein the feature interpretation module calculates the average marginal contribution of each feature in the sample by the following formula:
wherein the content of the first and second substances,
representing the average marginal contribution of the ith feature; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features of the sample; f. ofxA function representing the original model; f. ofx(S∪{i})-fx(S) represents the amount of increase in the contribution margin when the ith feature is added to the ranked subset S.
7. The apparatus for feature interpretation of deep learning calculation results according to claim 5 or 6, wherein the feature interpretation module is further configured to calculate the average marginal contribution of the intersection of the plurality of features in the sample by the following formula:
wherein the content of the first and second substances,
representing t featuresThe average marginal contribution of the intersection; n represents the set of global features of the sample; s represents a permutation subset of N; m represents the total number of features; n is a feature set composed of t features in N, fxA function representing the original model;
indicating the amount of increase in the contribution margin when t features in n are added to the ranked subset S.
8. The apparatus for feature interpretation of deep learning calculation results according to claim 5 or 6, wherein the feature interpretation module is further configured to calculate the evaluation model value of each feature according to the average marginal contribution of each feature in each sample, wherein the variation trend of the evaluation model value is proportional to the positive and negative of the influence of each feature on the calculation results of the original model output.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for feature interpretation of deep learning computation results of any one of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium characterized by storing a computer program for executing the feature interpretation method of the deep learning calculation result according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110178696.7A CN112784986A (en) | 2021-02-08 | 2021-02-08 | Feature interpretation method, device, equipment and medium for deep learning calculation result |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110178696.7A CN112784986A (en) | 2021-02-08 | 2021-02-08 | Feature interpretation method, device, equipment and medium for deep learning calculation result |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112784986A true CN112784986A (en) | 2021-05-11 |
Family
ID=75761403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110178696.7A Pending CN112784986A (en) | 2021-02-08 | 2021-02-08 | Feature interpretation method, device, equipment and medium for deep learning calculation result |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112784986A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113570260A (en) * | 2021-07-30 | 2021-10-29 | 北京房江湖科技有限公司 | Task allocation method, computer-readable storage medium and electronic device |
| CN113723618A (en) * | 2021-08-27 | 2021-11-30 | 南京星环智能科技有限公司 | SHAP optimization method, equipment and medium |
| CN115238596A (en) * | 2022-09-22 | 2022-10-25 | 中科三清科技有限公司 | Data processing method and device, readable storage medium and electronic equipment |
| CN115579092A (en) * | 2022-11-25 | 2023-01-06 | 安徽瑞邦数科科技服务有限公司 | Method and system for predicting contribution value and application of method and system in phosphoric acid production |
| CN116704208A (en) * | 2023-08-04 | 2023-09-05 | 南京理工大学 | Local interpretable method based on characteristic relation |
| CN117710066A (en) * | 2024-02-05 | 2024-03-15 | 厦门傲凡科技股份有限公司 | Financial customer recommendation method and system |
-
2021
- 2021-02-08 CN CN202110178696.7A patent/CN112784986A/en active Pending
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113570260A (en) * | 2021-07-30 | 2021-10-29 | 北京房江湖科技有限公司 | Task allocation method, computer-readable storage medium and electronic device |
| CN113723618A (en) * | 2021-08-27 | 2021-11-30 | 南京星环智能科技有限公司 | SHAP optimization method, equipment and medium |
| CN115238596A (en) * | 2022-09-22 | 2022-10-25 | 中科三清科技有限公司 | Data processing method and device, readable storage medium and electronic equipment |
| CN115238596B (en) * | 2022-09-22 | 2023-01-31 | 中科三清科技有限公司 | Data processing method and device, readable storage medium and electronic equipment |
| CN115579092A (en) * | 2022-11-25 | 2023-01-06 | 安徽瑞邦数科科技服务有限公司 | Method and system for predicting contribution value and application of method and system in phosphoric acid production |
| CN115579092B (en) * | 2022-11-25 | 2023-04-07 | 安徽瑞邦数科科技服务有限公司 | Contribution value prediction method and system and application thereof in phosphoric acid production |
| CN116704208A (en) * | 2023-08-04 | 2023-09-05 | 南京理工大学 | Local interpretable method based on characteristic relation |
| CN116704208B (en) * | 2023-08-04 | 2023-10-20 | 南京理工大学 | Local interpretable method based on characteristic relation |
| CN117710066A (en) * | 2024-02-05 | 2024-03-15 | 厦门傲凡科技股份有限公司 | Financial customer recommendation method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112784986A (en) | Feature interpretation method, device, equipment and medium for deep learning calculation result | |
| Kwilinski et al. | Information support of the entrepreneurship model complex with the application of cloud technologies | |
| US20180308160A1 (en) | Risk assessment method and system | |
| US11250499B2 (en) | Generating optimal strategy for providing offers | |
| US8706596B2 (en) | Account portfolio risk characterization | |
| Fritz-Morgenthal et al. | Financial risk management and explainable, trustworthy, responsible AI | |
| Roy et al. | A credit scoring model for SMEs using AHP and TOPSIS | |
| Ghahramanzadeh | Managing risk of construction projects: A case study of Iran | |
| US20110153536A1 (en) | Computer-Implemented Systems And Methods For Dynamic Model Switching Simulation Of Risk Factors | |
| US20090177612A1 (en) | Method and Apparatus for Analyzing Data to Provide Decision Making Information | |
| CN111340616A (en) | Method, device, equipment and medium for approving online loan | |
| CN110472815A (en) | To the risk control method and system of financing enterprise in a kind of supply chain financial business | |
| US20220004923A1 (en) | Systems and methods for model explanation | |
| Wong | Bubble value at risk: A countercyclical risk management approach | |
| Swankie et al. | Examining the impact of artificial intelligence on the evaluation of banking risk | |
| CN114846500A (en) | System for performing selections from dynamically generated electronic databases | |
| US20110264602A1 (en) | Computer-Implemented Systems And Methods For Implementing Dynamic Trading Strategies In Risk Computations | |
| CN115809837A (en) | Financial enterprise management method, equipment and medium based on digital simulation scene | |
| Hašková et al. | Internal rate of return estimation of subsidised projects: conventional approach versus fuzzy approach | |
| Theuri et al. | The impact of Artficial Intelligence and how it is shaping banking | |
| Kuzmina et al. | Risk Analysis of the Company's Activities by Means of Simulation. | |
| Ruzgar et al. | Rough sets and logistic regression analysis for loan payment | |
| CN111328406A (en) | System and method for automatically preparing visual representation of target realizability | |
| Qiu | Estimation of tail risk measures in finance: Approaches to extreme value mixture modeling | |
| Veynberg et al. | Intelligent simulation models based on business rules approach in banking sector (WIP). |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |