CN118551853A - Method, device, storage medium and electronic equipment for explaining model - Google Patents
Method, device, storage medium and electronic equipment for explaining model Download PDFInfo
- Publication number
- CN118551853A CN118551853A CN202411025269.5A CN202411025269A CN118551853A CN 118551853 A CN118551853 A CN 118551853A CN 202411025269 A CN202411025269 A CN 202411025269A CN 118551853 A CN118551853 A CN 118551853A
- Authority
- CN
- China
- Prior art keywords
- model
- time step
- linear
- neural network
- proxy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 93
- 238000003062 neural network model Methods 0.000 claims abstract description 102
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 69
- 238000012549 training Methods 0.000 claims abstract description 59
- 230000008569 process Effects 0.000 claims abstract description 25
- 239000003795 chemical substances by application Substances 0.000 claims description 88
- 230000000306 recurrent effect Effects 0.000 claims description 36
- 239000013598 vector Substances 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 18
- 230000006399 behavior Effects 0.000 claims description 10
- 238000012417 linear regression Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000006872 improvement Effects 0.000 description 8
- 238000013461 design Methods 0.000 description 6
- 238000013136 deep learning model Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 101100134058 Caenorhabditis elegans nth-1 gene Proteins 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007500 overflow downdraw method Methods 0.000 description 2
- 235000008733 Citrus aurantifolia Nutrition 0.000 description 1
- 235000011941 Tilia x europaea Nutrition 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 239000004571 lime Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Machine Translation (AREA)
Abstract
The embodiment of the specification discloses a method, a device, a storage medium and electronic equipment for explaining a model, which are used for inputting text sequence information into a cyclic neural network model to be explained, constructing a linear proxy model corresponding to each time step of the text sequence information in the input process according to the behavior of the cyclic neural network model in a prediction process corresponding to the time step, and training the linear proxy model to obtain a trained linear proxy model; constructing an integrated agent model according to the trained linear agent model corresponding to each time step; and obtaining feature attribution scores corresponding to the text units of each time step based on the integrated agent model according to the output result of the cyclic neural network model about the text sequence information, and obtaining interpretation information for interpreting the cyclic neural network model according to the feature attribution scores.
Description
Technical Field
The present invention relates to computer technology, and in particular, to a method, an apparatus, a storage medium, and an electronic device for interpreting a model.
Background
In the prior art, since the prediction result of the deep learning model has a black box characteristic, it is difficult for a user to understand the decision mechanism and reason thereof from the model prediction, and to intuitively and easily understand the interpretation, so that an interpretable algorithm needs to be introduced to explicitly represent the internal logic relationship existing between the prediction result of the model and the corresponding sample.
Task data in a sequence scene can provide richer context semantic information to help a model to make a decision, but has the characteristics of high feature dimension, complex data form and content, and the service scene actually applied is complex and changeable due to the time dimension, so that the existing machine learning interpretable method is difficult to directly apply. The cyclic neural network model is a model which is commonly used in a real scene and is used for processing sequence data, and can achieve a good prediction effect by using a small parameter quantity and a relatively simple structure. Therefore, an effective interpretable method is required to provide an explanation for the prediction of the model for the prediction of the recurrent neural network model in the sequence scene, and help understand the decision logic of the model in the scene.
Disclosure of Invention
An object of embodiments of the present specification is to provide a method, an apparatus, a storage medium, and an electronic device for interpreting a model.
The embodiment of the specification provides a method for explaining a model, and a special interpretable method which is obtained by characteristic design is carried out aiming at a specific structure and a prediction mechanism of a deep learning model which is widely applied to various service scenes such as a cyclic neural network model, the method integrates the specific structural characteristics of the cyclic neural network model into the design of the interpretable method by constructing an integrated proxy model, the prediction mechanism in the cyclic neural network is explicitly used in the modeling of the interpretable method, compared with the existing general interpretable method, the prediction mechanism of the model can be better helped by a user, the input feature is calculated, and the contribution and the influence of a hidden layer in the cyclic neural network model can be considered at the same time, so that the contribution and the influence of the hidden layer in model prediction are more visual and reasonable feature attributive interpretation can be generated, and the attributive interpretation which is more in line with the thinking mode of the user is better helped by the user, and the method comprises the following steps:
Inputting text sequence information into a cyclic neural network model to be interpreted, constructing a linear proxy model corresponding to a time step according to the behavior of the cyclic neural network model in a prediction process corresponding to the time step aiming at each time step of the text sequence information in the input process, and training the linear proxy model to obtain a trained linear proxy model, wherein the input of the linear proxy model is a text unit corresponding to the time step of the text sequence information and a current hidden state vector of the cyclic neural network model in the last time step of the time step, and the output of the linear proxy model is a predicted hidden state vector of the cyclic neural network model in the time step;
constructing an integrated agent model according to the trained linear agent model corresponding to each time step;
and obtaining feature attribution scores corresponding to the text units of each time step based on the integrated agent model according to the output result of the cyclic neural network model about the text sequence information, and obtaining interpretation information for interpreting the cyclic neural network model according to the feature attribution scores.
Further, the training the linear proxy model to obtain a trained linear proxy model includes:
Constructing a training sample according to the input characteristics of the linear agent model;
And training the linear proxy model according to the training sample to obtain a trained linear proxy model.
Further, the constructing a training sample according to the input features of the linear proxy model includes:
And obtaining a training sample by applying random disturbance to the input features of the linear proxy model.
Further, the training samples are distributed in adjacent spaces of the input features of the linear proxy model.
Further, the training the linear proxy model according to the training sample to obtain a trained linear proxy model includes:
And fitting the linear proxy model based on a linear regression algorithm according to the training sample and the current hidden state vector of the cyclic neural network model after the time step to obtain a trained linear proxy model.
Further, the constructing an integrated agent model according to the trained linear agent model corresponding to each time step includes:
And splicing the trained linear agent models corresponding to each time step end to end in sequence according to the time sequence to obtain an integrated agent model.
Further, the obtaining, based on the integrated proxy model, the feature-attribute score corresponding to the text unit of each time step according to the output result of the recurrent neural network model about the text sequence information includes:
Taking the output result of the recurrent neural network model about the text sequence information as an initial feature attribution score, and back-propagating the initial feature attribution score to the integrated proxy model to obtain feature attribution scores corresponding to the text units of each time step.
Further, the step of back-propagating the initial feature attribution score to the integrated proxy model to obtain feature attribution scores corresponding to the text units of each time step includes:
And reversely transmitting the initial feature attribution score to the integrated agent model, so that the initial feature attribution score is reversely transmitted to the last linear agent model in the integrated agent model, obtaining the feature attribution score corresponding to the text unit of the time step corresponding to the last linear agent model, continuously reversely transmitting the feature attribution score to the previous linear agent model corresponding to the last linear agent model, and the like until the feature attribution score corresponding to the text unit of the time step corresponding to the first linear agent model in the integrated agent model is obtained.
The embodiment of the specification also provides a device for explaining the model, which comprises:
The agent model training module is used for inputting text sequence information into a cyclic neural network model to be interpreted, constructing a linear agent model corresponding to a time step according to the behavior of the cyclic neural network model in a prediction process corresponding to the time step aiming at each time step of the text sequence information in the input process, and training the linear agent model to obtain a trained linear agent model, wherein the input of the linear agent model is a text unit corresponding to the time step of the text sequence information and a current hidden layer vector of the cyclic neural network model in the last time step corresponding to the time step, and the output of the linear agent model is a predicted hidden layer vector of the cyclic neural network model in the time step;
The integrated agent model building module is used for building an integrated agent model according to the trained linear agent model corresponding to each time step;
And the model interpretation module is used for obtaining feature attribution scores corresponding to the text units of each time step based on the integrated agent model according to the output result of the cyclic neural network model about the text sequence information, and obtaining interpretation information for interpreting the cyclic neural network model according to the feature attribution scores.
The present description also provides a computer program product storing at least one instruction adapted to be loaded by a processor and to perform the above-described method steps.
The present description embodiment also provides a storage medium storing a computer program adapted to be loaded by a processor and to perform the steps of the above-described method.
The embodiment of the specification also provides an electronic device, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the steps of the method described above.
In the embodiment of the specification, aiming at the special structure and prediction mechanism of the deep learning model which is widely applied in various service scenes, such as the cyclic neural network model, a special interpretable method is performed, which is characterized in that the special structure characteristics of the cyclic neural network model are integrated into the design of the interpretable method by constructing an integrated proxy model, the prediction mechanism in the cyclic neural network is explicitly used in the modeling of the interpretable method, compared with the existing general interpretable method, the prediction mechanism of the model can be better helped by a user, the input feature is calculated, and the contribution and influence of the hidden state in the cyclic neural network model can be considered at the same time, so that the contribution and influence in model prediction are more visual and reasonable feature attribution interpretation can be provided, attribution interpretation which is more in line with the thinking mode of the user can be generated, and the user can be helped to better understand the prediction mechanism of the cyclic neural network model.
Drawings
FIG. 1 is a flow chart of a method for explaining a model according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of an exemplary integrated proxy model provided by embodiments of the present disclosure;
FIG. 3 is a flow diagram of an exemplary method for integrated agent model back propagation provided by embodiments of the present description;
fig. 4 is a schematic structural view of an apparatus for explaining a model according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the technical solutions of the present specification will be clearly and completely described below with reference to specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present specification. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
Referring to fig. 1, a flow chart of a method for explaining a model is provided in an embodiment of the present disclosure. In the present embodiment, the method for interpreting a model is applied to an apparatus for interpreting a model (hereinafter simply referred to as "model interpretation apparatus") or an electronic device provided with the model interpretation apparatus. The following will describe the flow shown in fig. 1 in detail, and the method for explaining the model may specifically include the following steps:
S102, inputting text sequence information into a cyclic neural network model to be interpreted, constructing a linear proxy model corresponding to a time step according to the behavior of the cyclic neural network model in a prediction process corresponding to the time step aiming at each time step of the text sequence information in the input process, and training the linear proxy model to obtain a trained linear proxy model, wherein the input of the linear proxy model is a text unit corresponding to the time step of the text sequence information and a current hidden state vector of the cyclic neural network model in the last time step of the time step, and the output of the linear proxy model is a predicted hidden state vector of the cyclic neural network model in the time step.
In some embodiments, the text sequence information includes a plurality of text elements (token) arranged in tandem. In some embodiments, the recurrent neural network model is a deep learning model commonly used when processing sequence data, and the memorization and forgetting of information in the sequence data are realized by updating vectors with hidden layer states in the model, wherein the hidden layer states are a core element in the recurrent neural network model, and when the model processes input data, the state vectors are updated time step by time step to acquire information in input characteristics, and the part of information to be memorized or forgotten is determined. In some embodiments, the recurrent neural network model in the present exemplary embodiment may be any recurrent neural network model used to predict text tasks including, but not limited to, emotion analysis tasks, news classification tasks, etc., which the present exemplary embodiment does not specifically limit.
In some embodiments, during the inputting of the text sequence information into the recurrent neural network model to be interpreted, the text units in the text sequence information are sequentially input into the recurrent neural network model in time steps (i.e. time intervals) based on their arrangement in the text sequence information, i.e. at least one text unit in the text sequence information is input into the recurrent neural network model in each time step, e.g. the first N text units in the text sequence information are input into the recurrent neural network model in the first time step, M text units in the text sequence information arranged after the N text units are input into the recurrent neural network model in the second time step, P text units in the text sequence information arranged after the M text units are input into the recurrent neural network model in the third time step, and so on until all text units in the text sequence information have been input into the recurrent neural network model. In some embodiments, each time step corresponds to the same time interval, and the number or length of text units input into the recurrent neural network model in each time step may be random or may be fixed by a preset setting, which is not particularly limited in this exemplary embodiment.
In some embodiments, for each time step in the input process of the text sequence information, a linear proxy model corresponding to the time step is constructed according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step, where the behavior data includes, but is not limited to, any behavior (e.g., internal computing logic, etc.) executed by the recurrent neural network model in the time step, and the present exemplary embodiment does not make special restrictions on this, or constructs a linear proxy model corresponding to the time step according to the behavior of the recurrent neural network model in the prediction process corresponding to the time step and the text unit corresponding to the time step of the text sequence information (i.e., the text unit in the text sequence information that is input to the recurrent neural network model in the time step). In some embodiments, each time step corresponds to a linear proxy model, for each time step, the linear proxy model is input as a text unit corresponding to the text sequence information at the time step and a current hidden state vector of the cyclic neural network model after the last time step of the time step (the cyclic neural network updates the hidden state vector from time step to time step), and the output of the linear proxy model is a predicted hidden state vector of the cyclic neural network model after the time step, that is, the linear proxy model is used for predicting the updated latest hidden state vector of the cyclic neural network model after the time step. In some embodiments, the linear proxy model corresponding to each time step is trained to obtain a trained linear proxy model, and it should be understood by those skilled in the art that any training method may be included in the protection scope of the present specification, which is not limited in this specification
S104, constructing an integrated agent model according to the trained linear agent model corresponding to each time step. In some embodiments, the trained linear proxy model corresponding to each time step may be fused (integrated) into a proxy model through a preset model fusion method, and it should be understood by those skilled in the art that any model fusion method may be included in the protection scope of the present specification, which is not limited in this specification. In some embodiments, the trained linear proxy models corresponding to each time step may be combined together according to a certain rule to construct an integrated proxy model, and it should be understood by those skilled in the art that any combination rule may be included in the protection scope of the present specification, which is not limited in this specification.
S106, according to the output result of the cyclic neural network model about the text sequence information, based on the integrated agent model, obtaining feature attribution scores corresponding to the text units of each time step, and according to the feature attribution scores, obtaining interpretation information for interpreting the cyclic neural network model.
In some embodiments, an output result (i.e., a predicted result) of the recurrent neural network model with respect to the text sequence information is obtained, and based on the output result and based on the integrated proxy model obtained previously, a feature attribution score corresponding to the text unit of each time step is obtained, wherein the feature attribution score corresponding to each time step is used to characterize a degree of "importance" of the predicted result output by the neural network model for the recurrent neural network model for the text unit corresponding to the time step (i.e., the text unit input to the recurrent neural network model in the time step). The feature attribution score is used for representing the influence condition of different input features on model prediction by explicitly calculating the causal relation between the input features (namely, text units input into the cyclic neural network model in each time step) and the prediction result of the cyclic neural network model.
In some embodiments, a specific manner of obtaining the feature-attribute score corresponding to each time step based on the integrated proxy model may be to input an output result (i.e., a prediction result) of the recurrent neural network model about the text sequence information into the integrated proxy model, to obtain the feature-attribute score corresponding to each time step output by the integrated proxy model, or may also be to back-propagate the prediction result in the integrated proxy model, so as to obtain the feature-attribute score corresponding to each time step in the back-propagation process. In some embodiments, the interpretation information for interpreting the recurrent neural network model may be determined according to the feature-attribute scores corresponding to the respective time steps, for example, the feature-attribute scores corresponding to the respective time steps may be spliced together as the interpretation information for interpreting the recurrent neural network model, or the interpretation information for interpreting the recurrent neural network model may be determined according to the feature-attribute scores corresponding to the respective time steps and the text sequence information and/or the prediction result of the recurrent neural network model with respect to the text sequence information. In some embodiments, the interpretation information can provide interpretation for the prediction result of the cyclic neural network model, so that a user can understand the principle and mechanism of the decision of the cyclic neural network model, and the prediction result is transparent and interpretable.
The embodiment of the embodiment carries out a special interpretable method which is obtained by characteristic design aiming at the special structure and prediction mechanism of a deep learning model which is widely applied in various service scenes, such as a cyclic neural network model, the method integrates the special structural characteristics of the cyclic neural network model into the design of the interpretable method by constructing an integrated proxy model, the prediction mechanism inside the cyclic neural network is explicitly used in the modeling of the interpretable method, compared with the existing general interpretable method, the method can better help users understand the prediction mechanism of the model, calculate the attribution score for the input characteristics, and simultaneously consider the contribution and influence of the core element in the hidden layer type cyclic neural network model in model prediction, thereby providing more visual and reasonable characteristic attribution interpretation, generating attribution interpretation which better accords with the thinking mode of the users, and helping the users to better understand the prediction mechanism of the cyclic neural network model.
In some embodiments, the training the linear proxy model to obtain a trained linear proxy model includes: constructing a training sample according to the input characteristics of the linear agent model; and training the linear proxy model according to the training sample to obtain a trained linear proxy model. In some embodiments, for the linear proxy model corresponding to each time step, according to the input feature of the linear proxy model, that is, the text unit corresponding to the time step (the text unit of the cyclic neural network model is input in the time step), a plurality of training samples corresponding to the linear proxy model are constructed, and the linear proxy model is trained through the plurality of training samples, so as to obtain a trained linear proxy model, for example, a preset data enhancement mode may be adopted to perform data enhancement processing on the text unit, so as to obtain a plurality of enhanced text units corresponding to the text unit and use the text units as training samples corresponding to the linear proxy model.
In some embodiments, the constructing a training sample according to the input features of the linear proxy model includes: and obtaining a training sample by applying random disturbance to the input features of the linear proxy model. In some embodiments, by referring to the LIME interpretable method, for each time step corresponding linear proxy model, a random perturbation is applied to the input features of the linear proxy model, i.e., the text units corresponding to that time step (the text units of the recurrent neural network model were input during that time step), to obtain a plurality of perturbation samples as training samples for the linear proxy model.
In some embodiments, the training samples are distributed in adjacent spaces of the input features of the linear proxy model. In some embodiments, the disturbance samples obtained by applying random disturbance to the input features of the linear proxy model are distributed in the adjacent space of the input features of the linear proxy model, so that the linear proxy model can be helped to learn the prediction logic of the recurrent neural network model at the current time step corresponding to the linear proxy model.
In some embodiments, the training the linear proxy model according to the training sample to obtain a trained linear proxy model includes: and fitting the linear proxy model based on a linear regression algorithm according to the training sample and the current hidden state vector of the cyclic neural network model after the time step to obtain a trained linear proxy model. In some embodiments, the updated and updated actual hidden state vector of the cyclic neural network model after the time step is obtained, the actual hidden state vector is used as label information corresponding to a training sample, and the linear agent model is fitted (i.e. regression training is performed on the linear agent model) based on a linear regression algorithm according to the training sample and the label information corresponding to the training sample, so as to obtain a trained linear agent model.
In some embodiments, the constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step includes: and splicing the trained linear agent models corresponding to each time step end to end in sequence according to the time sequence to obtain an integrated agent model. In some embodiments, the trained linear proxy models corresponding to each time step are spliced sequentially end to end according to the time sequence corresponding to the time step, so as to obtain a spliced integrated proxy model, the output of the former linear proxy model is the output of the latter linear proxy model, the input of the first linear proxy model is the output of the integrated proxy model, the output of the last linear proxy model is the output of the integrated proxy model, that is, the output of the trained linear proxy model corresponding to a certain time step in time sequence is the input of the trained linear proxy model corresponding to the next time step corresponding to the time step, and the input of the trained linear proxy model corresponding to a certain time step in time sequence is the output of the trained linear proxy model corresponding to the last time step corresponding to the time step. As an example, as shown in fig. 2, the input of the proxy model 1 corresponding to the first time step is the input of the integrated proxy model, the output of the proxy model 1 is the input of the proxy model 2 corresponding to the second time step, and so on, the output of the proxy model corresponding to the previous time step is the input of the proxy model corresponding to the next time step, and the output of the nth time step (i.e., the last time step) is the output of the integrated proxy model.
In some embodiments, the obtaining, based on the integrated proxy model, the feature-attribute score corresponding to the text unit of each time step according to the output result of the recurrent neural network model about the text sequence information includes: taking the output result of the recurrent neural network model about the text sequence information as an initial feature attribution score, and back-propagating the initial feature attribution score to the integrated proxy model to obtain feature attribution scores corresponding to the text units of each time step. In some embodiments, the output result (i.e. the prediction result) of the recurrent neural network model about the text sequence information is taken as an initial feature attribution score, the initial feature attribution score is back-propagated in the integrated proxy model, that is, the initial feature attribution score is transferred to each linear proxy model inside the integrated proxy model according to a back-propagation rule, and the feature attribution score corresponding to each time step is obtained in the back-propagation process.
In some embodiments, the back-propagating the initial feature-attribute score to the integrated proxy model, obtaining the feature-attribute score corresponding to the text unit for each time step includes: and reversely transmitting the initial feature attribution score to the integrated agent model, so that the initial feature attribution score is reversely transmitted to the last linear agent model in the integrated agent model, obtaining the feature attribution score corresponding to the text unit of the time step corresponding to the last linear agent model, continuously reversely transmitting the feature attribution score to the previous linear agent model corresponding to the last linear agent model, and the like until the feature attribution score corresponding to the text unit of the time step corresponding to the first linear agent model in the integrated agent model is obtained. In some embodiments, the initial feature-attribute score is back-propagated to the integrated proxy model such that the initial feature-attribute score is back-propagated to a last linear proxy model in the integrated proxy model, the feature-attribute score for the time step corresponding to the last linear proxy model is obtained during back-propagation of the initial feature-attribute score in the last linear proxy model, and the feature-attribute score is continued to be back-propagated to a previous linear proxy model corresponding to the last linear proxy model in the integrated proxy model, a new feature-attribute score for the time step corresponding to the previous linear proxy model is obtained during back-propagation of the feature-attribute score in the previous linear proxy model, and the new feature-attribute score is continued to be back-propagated until the feature-attribute score for the time step corresponding to the first linear proxy model in the integrated proxy model is obtained, and so on, to gradually pass the feature-attribute score from back-to-back in each linear proxy model within the integrated proxy model following a back-propagation rule. As an example, as shown in fig. 3, the initial feature-attribution score is back-propagated to the integrated proxy model, such that the initial feature-attribution score is back-propagated to the proxy model N corresponding to the nth time step (i.e., the last time step) in the integrated proxy model, the feature-attribution score N corresponding to the nth time step is obtained during the back-propagation, and such that the feature-attribution score N is continuously back-propagated to the proxy model N-1 corresponding to the nth-1 time step (i.e., the last-last time step) in the integrated proxy model, the feature-attribution score N-1 corresponding to the nth-1 time step is obtained during the back-propagation, and so on until the feature-attribution score 1 corresponding to the first time step in the integrated proxy model is obtained during the back-propagation of the proxy model 1 corresponding to the first time step.
Fig. 4 is a schematic structural diagram of an apparatus for interpreting a model (hereinafter, simply referred to as "model interpretation apparatus 1") that can be implemented as all or a part of an electronic device by software, hardware, or a combination of both, according to an embodiment of the present specification. According to some embodiments, the model interpretation apparatus 1 comprises a proxy model training module 11, an integrated proxy model building module 12 and a model interpretation module 13.
The agent model training module 11 is configured to input text sequence information into a cyclic neural network model to be interpreted, construct a linear agent model corresponding to a time step according to a behavior of the cyclic neural network model in a prediction process corresponding to the time step for each time step in the input process of the text sequence information, and train the linear agent model to obtain a trained linear agent model, where input of the linear agent model is a text unit corresponding to the time step of the text sequence information and a current hidden state vector of the cyclic neural network model in a last time step corresponding to the time step, and output of the linear agent model is a predicted hidden state vector of the cyclic neural network model in the time step;
an integrated agent model construction module 12, configured to construct an integrated agent model according to the trained linear agent model corresponding to each time step;
And the model interpretation module 13 is used for obtaining feature attribution scores corresponding to the text units of each time step based on the integrated proxy model according to the output result of the cyclic neural network model about the text sequence information, and obtaining interpretation information for interpreting the cyclic neural network model according to the feature attribution scores.
In some embodiments, the training the linear proxy model to obtain a trained linear proxy model includes:
Constructing a training sample according to the input characteristics of the linear agent model;
And training the linear proxy model according to the training sample to obtain a trained linear proxy model.
In some embodiments, the constructing a training sample according to the input features of the linear proxy model includes:
And obtaining a training sample by applying random disturbance to the input features of the linear proxy model.
In some embodiments, the training samples are distributed in adjacent spaces of the input features of the linear proxy model.
In some embodiments, the training the linear proxy model according to the training sample to obtain a trained linear proxy model includes:
And fitting the linear proxy model based on a linear regression algorithm according to the training sample and the current hidden state vector of the cyclic neural network model after the time step to obtain a trained linear proxy model.
In some embodiments, the constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step includes:
And splicing the trained linear agent models corresponding to each time step end to end in sequence according to the time sequence to obtain an integrated agent model.
In some embodiments, the obtaining, based on the integrated proxy model, the feature-attribute score corresponding to the text unit of each time step according to the output result of the recurrent neural network model about the text sequence information includes:
Taking the output result of the recurrent neural network model about the text sequence information as an initial feature attribution score, and back-propagating the initial feature attribution score to the integrated proxy model to obtain feature attribution scores corresponding to the text units of each time step.
In some embodiments, the back-propagating the initial feature-attribute score to the integrated proxy model, obtaining the feature-attribute score corresponding to the text unit for each time step includes:
And reversely transmitting the initial feature attribution score to the integrated agent model, so that the initial feature attribution score is reversely transmitted to the last linear agent model in the integrated agent model, obtaining the feature attribution score corresponding to the text unit of the time step corresponding to the last linear agent model, continuously reversely transmitting the feature attribution score to the previous linear agent model corresponding to the last linear agent model, and the like until the feature attribution score corresponding to the text unit of the time step corresponding to the first linear agent model in the integrated agent model is obtained.
The foregoing apparatus embodiments correspond to the method embodiments, and specific descriptions may be referred to descriptions of method embodiment portions, which are not repeated herein. The device embodiments are obtained based on corresponding method embodiments, and have the same technical effects as the corresponding method embodiments, and specific description can be found in the corresponding method embodiments.
The present description also provides a computer storage medium that may store a plurality of instructions adapted to be loaded by a processor and to perform the methods of the present description embodiments.
The present description embodiment also provides a computer program product storing at least one instruction for loading and executing the method of the present description embodiment by the processor.
The embodiment of the specification also provides a schematic structural diagram of the electronic device shown in fig. 5. At the hardware level, as in fig. 5, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile storage, although it may include hardware required for other services. The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to realize the voice activity detection method.
Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present description, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable GATE ARRAY, FPGA)) is an integrated circuit whose logic functions are determined by user programming of the device. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented with "logic compiler (logic compiler)" software, which is similar to the software compiler used in program development and writing, and the original code before being compiled is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but HDL is not just one, but a plurality of kinds, such as ABEL（Advanced Boolean Expression Language）、AHDL（Altera Hardware Description Language）、Confluence、CUPL（Cornell University Programming Language）、HDCal、JHDL（Java Hardware Description Language）、Lava、Lola、MyHDL、PALASM、RHDL（Ruby Hardware Description Language）, and VHDL (Very-High-SPEED INTEGRATED Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application SPECIFIC INTEGRATED Circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.
Claims (12)
1. A method for interpreting a model, comprising:
Inputting text sequence information into a cyclic neural network model to be interpreted, constructing a linear proxy model corresponding to a time step according to the behavior of the cyclic neural network model in a prediction process corresponding to the time step aiming at each time step of the text sequence information in the input process, and training the linear proxy model to obtain a trained linear proxy model, wherein the input of the linear proxy model is a text unit corresponding to the time step of the text sequence information and a current hidden state vector of the cyclic neural network model in the last time step of the time step, and the output of the linear proxy model is a predicted hidden state vector of the cyclic neural network model in the time step;
constructing an integrated agent model according to the trained linear agent model corresponding to each time step;
and obtaining feature attribution scores corresponding to the text units of each time step based on the integrated agent model according to the output result of the cyclic neural network model about the text sequence information, and obtaining interpretation information for interpreting the cyclic neural network model according to the feature attribution scores.
2. The method of claim 1, the training the linear proxy model to obtain a trained linear proxy model, comprising:
Constructing a training sample according to the input characteristics of the linear agent model;
And training the linear proxy model according to the training sample to obtain a trained linear proxy model.
3. The method of claim 2, the constructing training samples from input features of the linear proxy model, comprising:
And obtaining a training sample by applying random disturbance to the input features of the linear proxy model.
4. A method according to claim 3, wherein the training samples are distributed in adjacent spaces of the input features of the linear proxy model.
5. The method of claim 4, wherein the training the linear proxy model according to the training sample to obtain a trained linear proxy model comprises:
And fitting the linear proxy model based on a linear regression algorithm according to the training sample and the current hidden state vector of the cyclic neural network model after the time step to obtain a trained linear proxy model.
6. The method of claim 1, wherein the constructing an integrated proxy model according to the trained linear proxy model corresponding to each time step comprises:
And splicing the trained linear agent models corresponding to each time step end to end in sequence according to the time sequence to obtain an integrated agent model.
7. The method of claim 6, wherein the obtaining, based on the integrated proxy model, the feature-attribute score corresponding to the text unit of each time step according to the output result of the recurrent neural network model with respect to the text sequence information, comprises:
Taking the output result of the recurrent neural network model about the text sequence information as an initial feature attribution score, and back-propagating the initial feature attribution score to the integrated proxy model to obtain feature attribution scores corresponding to the text units of each time step.
8. The method of claim 7, the back-propagating the initial feature-attribution score to the integrated proxy model, obtaining the feature-attribution score for each time step's text element, comprising:
And reversely transmitting the initial feature attribution score to the integrated agent model, so that the initial feature attribution score is reversely transmitted to the last linear agent model in the integrated agent model, obtaining the feature attribution score corresponding to the text unit of the time step corresponding to the last linear agent model, continuously reversely transmitting the feature attribution score to the previous linear agent model corresponding to the last linear agent model, and the like until the feature attribution score corresponding to the text unit of the time step corresponding to the first linear agent model in the integrated agent model is obtained.
9. An apparatus for interpreting a model, comprising:
The agent model training module is used for inputting text sequence information into a cyclic neural network model to be interpreted, constructing a linear agent model corresponding to a time step according to the behavior of the cyclic neural network model in a prediction process corresponding to the time step aiming at each time step of the text sequence information in the input process, and training the linear agent model to obtain a trained linear agent model, wherein the input of the linear agent model is a text unit corresponding to the time step of the text sequence information and a current hidden layer vector of the cyclic neural network model in the last time step corresponding to the time step, and the output of the linear agent model is a predicted hidden layer vector of the cyclic neural network model in the time step;
The integrated agent model building module is used for building an integrated agent model according to the trained linear agent model corresponding to each time step;
And the model interpretation module is used for obtaining feature attribution scores corresponding to the text units of each time step based on the integrated agent model according to the output result of the cyclic neural network model about the text sequence information, and obtaining interpretation information for interpreting the cyclic neural network model according to the feature attribution scores.
10. A storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the method according to any of claims 1-8.
11. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the steps of the method according to any of claims 1-8.
12. A computer program product having stored thereon at least one instruction, which when executed by a processor, implements the steps of the method according to any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411025269.5A CN118551853A (en) | 2024-07-29 | 2024-07-29 | Method, device, storage medium and electronic equipment for explaining model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411025269.5A CN118551853A (en) | 2024-07-29 | 2024-07-29 | Method, device, storage medium and electronic equipment for explaining model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118551853A true CN118551853A (en) | 2024-08-27 |
Family
ID=92448643
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411025269.5A Pending CN118551853A (en) | 2024-07-29 | 2024-07-29 | Method, device, storage medium and electronic equipment for explaining model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118551853A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220147876A1 (en) * | 2020-11-12 | 2022-05-12 | UMNAI Limited | Architecture for explainable reinforcement learning |
| CN115221137A (en) * | 2022-06-24 | 2022-10-21 | 山东科技大学 | Flow log identification method based on gated recurrent neural network internal state interpretation |
| CN118070955A (en) * | 2024-02-21 | 2024-05-24 | 同济大学 | Vehicle real-time fuel quantity calculating method considering SHAP (SHAP-based analysis) XGBoost-FCGWA-KELM |
| WO2024120676A1 (en) * | 2022-12-05 | 2024-06-13 | Siemens Aktiengesellschaft | Local and global interpretable explanation machine learning model for explaining an output of an analysis model analyzing a machine |
-
2024
- 2024-07-29 CN CN202411025269.5A patent/CN118551853A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220147876A1 (en) * | 2020-11-12 | 2022-05-12 | UMNAI Limited | Architecture for explainable reinforcement learning |
| CN115221137A (en) * | 2022-06-24 | 2022-10-21 | 山东科技大学 | Flow log identification method based on gated recurrent neural network internal state interpretation |
| WO2024120676A1 (en) * | 2022-12-05 | 2024-06-13 | Siemens Aktiengesellschaft | Local and global interpretable explanation machine learning model for explaining an output of an analysis model analyzing a machine |
| CN118070955A (en) * | 2024-02-21 | 2024-05-24 | 同济大学 | Vehicle real-time fuel quantity calculating method considering SHAP (SHAP-based analysis) XGBoost-FCGWA-KELM |
Non-Patent Citations (3)
| Title |
|---|
| GUANYU JIANG等: "SeqSHAP:Subsequence Level Shapley Value Explanations for Sequential Predictions", ICLR 2023 CONFERENCE, 2 February 2023 (2023-02-02) * |
| LEILA ARRAS等: "Evaluating Recurrent Neural Network Explanations", ARXIV:1904.11829V3, 4 June 2019 (2019-06-04) * |
| 张新新: "基于隐马尔科夫模型的循环神经网络解释研究", 中国优秀硕士学位论文全文数据库, 15 January 2024 (2024-01-15) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN117406982B (en) | Integrated storage and calculation application generation system and method, storage medium and equipment | |
| CN115203394A (en) | Model training method, service execution method and device | |
| CN117828360A (en) | Model training method, model training device, model code generating device, storage medium and storage medium | |
| CN113343085B (en) | Information recommendation method and device, storage medium and electronic equipment | |
| CN117332282B (en) | Knowledge graph-based event matching method and device | |
| CN117409466B (en) | Three-dimensional dynamic expression generation method and device based on multi-label control | |
| CN116151355B (en) | Method, device, medium and equipment for model training and service execution | |
| CN116341642B (en) | Data processing method and device, storage medium and electronic equipment | |
| CN115017915B (en) | Model training and task execution method and device | |
| CN118551853A (en) | Method, device, storage medium and electronic equipment for explaining model | |
| CN113344590A (en) | Method and device for model training and complaint rate estimation | |
| CN116501852B (en) | Controllable dialogue model training method and device, storage medium and electronic equipment | |
| CN117786061B (en) | Large language model prediction method and device based on space-time attention mechanism | |
| CN117351946B (en) | Voice recognition method and device, storage medium and electronic equipment | |
| CN116996397B (en) | Network packet loss optimization method and device, storage medium and electronic equipment | |
| CN118468045B (en) | Model training acceleration method, device and storage medium | |
| CN117591217A (en) | Information display method, device, equipment and storage medium | |
| CN117593003A (en) | Model training method and device, storage medium and electronic equipment | |
| CN116070718A (en) | Data processing system based on machine learning | |
| CN118569392A (en) | Method, device, storage medium and electronic equipment for explaining model | |
| CN117992787A (en) | Model training method, task execution method and device | |
| CN117351946A (en) | Voice recognition method and device, storage medium and electronic equipment | |
| CN117171346A (en) | Entity linking method and device, storage medium and electronic equipment | |
| CN117034926A (en) | Model training method and device for multi-field text classification model | |
| CN118098266A (en) | Voice data processing method and device based on multi-model selection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |