"Adapted for AI-Campus course [XAI4Ing](https://learn.ki-campus.org/courses/erklaerbareki2020) by Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/\n",
"\n",
"\n",
"## Trepan\n",
"\n",
"[M. Craven and J. Shalvik. 1995. Extracting tree-structured representations of trained networks. In Proceedings of the 8th International Conference on Neural Information Processing 1995](https://dl.acm.org/doi/10.5555/2998828.2998832)\n",
...
...
@@ -23,7 +27,16 @@
"\n",
"In this notebook we will use a modern implementation in the [`generalizedtrees`](https://github.com/Craven-Biostat-Lab/generalizedtrees) package to run Trepan on one of the original datasets used in the paper. `generalizedtrees` is under development with major changes from version to version. It is a project that attempts to bring together many different variants of tree learning together into a single framework.\n",
"\n",
"For this notebook we will use `generalizedtrees` version 1.1.0"
"For this notebook we will use `generalizedtrees` version 1.1.0\n",
"\n",
"\n",
"## Contact\n",
"Via the gitlab website: Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/\n",
"\n",
"## License\n",
"\n",
"This work is licensed under Creative Commons Attribution 4.0 International License.\n",
"https://creativecommons.org/licenses/by/4.0/\n"
]
},
{
...
...
%% Cell type:markdown id: tags:
# Using Trepan: a demonstration
By: Yuriy Sverchkov
Date: Feb 2021
Assistant Scientist
Department of Biostatistics and Medical Informatics
University of Wisconsin - Madison, USA
Adapted for AI-Campus course [XAI4Ing](https://learn.ki-campus.org/courses/erklaerbareki2020) by Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/
## Trepan
[M. Craven and J. Shalvik. 1995. Extracting tree-structured representations of trained networks. In Proceedings of the 8th International Conference on Neural Information Processing 1995](https://dl.acm.org/doi/10.5555/2998828.2998832)
Trepan is one of the first explanation-by-model-translation methods for Neural Networks.
Model translation is an approach to model explanation in which an uninterpretable black-box model is translated into an interpretable model (in this case a decision tree) that aims at having high fidelity to the black-box model, that is, make similar prediction to it.
In this notebook we will use a modern implementation in the [`generalizedtrees`](https://github.com/Craven-Biostat-Lab/generalizedtrees) package to run Trepan on one of the original datasets used in the paper. `generalizedtrees` is under development with major changes from version to version. It is a project that attempts to bring together many different variants of tree learning together into a single framework.
For this notebook we will use `generalizedtrees` version 1.1.0
## Contact
Via the gitlab website: Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/
## License
This work is licensed under Creative Commons Attribution 4.0 International License.
Our new feature set is below. Subsequently features will be referenced by their 0-index
%% Cell type:code id: tags:
``` python
pd.DataFrame({'Feature Name':feature_names})
```
%%%% Output: execute_result
Feature Name
0 age
1 resting bp
2 cholesterol
3 max heart rate
4 oldpeak
5 number of vessels colored
6 sex_male
7 chest pain type_abnang
8 chest pain type_angina
9 chest pain type_asympt
10 chest pain type_notang
11 fasting blood sugar < 120_True
12 resting ecg_abn
13 resting ecg_hyp
14 resting ecg_norm
15 exercise induced angina_True
16 slope_down
17 slope_flat
18 slope_up
19 thal_fix
20 thal_norm
21 thal_rev
%% Cell type:markdown id: tags:
## Black-box model
Like in the paper, our black-box model will be a fully connected Neural Network with one hidden layer, and the size of the layer is determined by cross-validation within the training set.
We have a function that serves as a 'recipe' for Trepan in our `generalizedtrees` python package, this returns an object that can be fit to data and a model.
%% Cell type:code id: tags:
``` python
explanation=trepan(
m_of_n=False,
max_tree_size=10,
impurity='entropy',
rng=np_rng)
```
%% Cell type:markdown id: tags:
We learn the explanation from the black-box model using the `fit` method, which takes unlabeled data and the black-box model as input.
%% Cell type:code id: tags:
``` python
t0=time.time()
explanation.fit(x_train,model)
t1=time.time()
print(f'Time taken to learn explanation: {t1-t0} seconds')
```
%%%% Output: stream
Assuming continuous features in the absence of feature specifications
%%%% Output: stream
Time taken to learn explanation: 3.302284002304077 seconds
%% Cell type:markdown id: tags:
A console-friendly representation of the tree is available:
%% Cell type:code id: tags:
``` python
print(explanation.show_tree())
```
%%%% Output: stream
Test x[5] > 0.5
+--Test x[5] > 1.5
| +--[0.49 0.51]
| +--[0.577 0.423]
+--Test x[4] > 1.55
+--[0.539 0.461]
+--Test x[20] > 0.5
+--Test x[4] > 1.45
| +--[0.561 0.439]
| +--[0.699 0.301]
+--[0.601 0.399]
%% Cell type:markdown id: tags:
A graphical representation (as an HTML file) can be generated as well:
Click the follwing link: [explanation.html](explanation.html)
%% Cell type:markdown id: tags:
## Performance comparison
The resulting tree can itself be treated as a classifier, and we can check how this learned tree performs on test data, alongside the original black-box model for comparison.
Another metric of interest for explanation methods specifically is *fidelity*, which measures how well the explanation's predictions match the black-box predictions:
There are some differences between the experiment presented here and the experiment in the paper. The version of Trepan we ran here did not use m-of-n splits at nodes (the search for splits becomes very slow if we enable it), we used a different number for the minimum samples required at each node (1000), and we used a simpler rejection-based sampling scheme for generating synthetic samples. The details of the neural network learning algorithm are also different.
This notebook is available [in a gist](https://gist.github.com/sverchkov/c87b301db1b88e0f4cc8bb7d77b889b9).
For issues/questions about the `generalizedtrees` package contact us through https://github.com/Craven-Biostat-Lab/generalizedtrees
"Adapted for AI-Campus course [XAI4Ing](https://learn.ki-campus.org/courses/erklaerbareki2020) by Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/\n",
"\n",
"\n",
"## Trepan\n",
"\n",
"[M. Craven and J. Shalvik. 1995. Extracting tree-structured representations of trained networks. In Proceedings of the 8th International Conference on Neural Information Processing 1995](https://dl.acm.org/doi/10.5555/2998828.2998832)\n",
...
...
@@ -23,7 +27,16 @@
"\n",
"In this notebook we will use a modern implementation in the [`generalizedtrees`](https://github.com/Craven-Biostat-Lab/generalizedtrees) package to run Trepan on one of the original datasets used in the paper. `generalizedtrees` is under development with major changes from version to version. It is a project that attempts to bring together many different variants of tree learning together into a single framework.\n",
"\n",
"For this notebook we will use `generalizedtrees` version 1.1.0"
"For this notebook we will use `generalizedtrees` version 1.1.0\n",
"\n",
"\n",
"## Contact\n",
"Via the gitlab website: Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/\n",
"\n",
"## License\n",
"\n",
"This work is licensed under Creative Commons Attribution 4.0 International License.\n",
"https://creativecommons.org/licenses/by/4.0/\n"
]
},
{
...
...
%% Cell type:markdown id: tags:
# Using Trepan: a demonstration
By: Yuriy Sverchkov
Date: Feb 2021
Assistant Scientist
Department of Biostatistics and Medical Informatics
University of Wisconsin - Madison, USA
Adapted for AI-Campus course [XAI4Ing](https://learn.ki-campus.org/courses/erklaerbareki2020) by Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/
## Trepan
[M. Craven and J. Shalvik. 1995. Extracting tree-structured representations of trained networks. In Proceedings of the 8th International Conference on Neural Information Processing 1995](https://dl.acm.org/doi/10.5555/2998828.2998832)
Trepan is one of the first explanation-by-model-translation methods for Neural Networks.
Model translation is an approach to model explanation in which an uninterpretable black-box model is translated into an interpretable model (in this case a decision tree) that aims at having high fidelity to the black-box model, that is, make similar prediction to it.
In this notebook we will use a modern implementation in the [`generalizedtrees`](https://github.com/Craven-Biostat-Lab/generalizedtrees) package to run Trepan on one of the original datasets used in the paper. `generalizedtrees` is under development with major changes from version to version. It is a project that attempts to bring together many different variants of tree learning together into a single framework.
For this notebook we will use `generalizedtrees` version 1.1.0
## Contact
Via the gitlab website: Stephan Scheele, https://www.uni-bamberg.de/en/cogsys/scheele-stephan/
## License
This work is licensed under Creative Commons Attribution 4.0 International License.
Our new feature set is below. Subsequently features will be referenced by their 0-index
%% Cell type:code id: tags:
``` python
pd.DataFrame({'Feature Name':feature_names})
```
%%%% Output: execute_result
Feature Name
0 age
1 resting bp
2 cholesterol
3 max heart rate
4 oldpeak
5 number of vessels colored
6 sex_male
7 chest pain type_abnang
8 chest pain type_angina
9 chest pain type_asympt
10 chest pain type_notang
11 fasting blood sugar < 120_True
12 resting ecg_abn
13 resting ecg_hyp
14 resting ecg_norm
15 exercise induced angina_True
16 slope_down
17 slope_flat
18 slope_up
19 thal_fix
20 thal_norm
21 thal_rev
%% Cell type:markdown id: tags:
## Black-box model
Like in the paper, our black-box model will be a fully connected Neural Network with one hidden layer, and the size of the layer is determined by cross-validation within the training set.
We have a function that serves as a 'recipe' for Trepan in our `generalizedtrees` python package, this returns an object that can be fit to data and a model.
%% Cell type:code id: tags:
``` python
explanation=trepan(
m_of_n=False,
max_tree_size=10,
impurity='entropy',
rng=np_rng)
```
%% Cell type:markdown id: tags:
We learn the explanation from the black-box model using the `fit` method, which takes unlabeled data and the black-box model as input.
%% Cell type:code id: tags:
``` python
t0=time.time()
explanation.fit(x_train,model)
t1=time.time()
print(f'Time taken to learn explanation: {t1-t0} seconds')
```
%%%% Output: stream
Assuming continuous features in the absence of feature specifications
%%%% Output: stream
Time taken to learn explanation: 3.302284002304077 seconds
%% Cell type:markdown id: tags:
A console-friendly representation of the tree is available:
%% Cell type:code id: tags:
``` python
print(explanation.show_tree())
```
%%%% Output: stream
Test x[5] > 0.5
+--Test x[5] > 1.5
| +--[0.49 0.51]
| +--[0.577 0.423]
+--Test x[4] > 1.55
+--[0.539 0.461]
+--Test x[20] > 0.5
+--Test x[4] > 1.45
| +--[0.561 0.439]
| +--[0.699 0.301]
+--[0.601 0.399]
%% Cell type:markdown id: tags:
A graphical representation (as an HTML file) can be generated as well:
Click the follwing link: [explanation.html](explanation.html)
%% Cell type:markdown id: tags:
## Performance comparison
The resulting tree can itself be treated as a classifier, and we can check how this learned tree performs on test data, alongside the original black-box model for comparison.
Another metric of interest for explanation methods specifically is *fidelity*, which measures how well the explanation's predictions match the black-box predictions:
There are some differences between the experiment presented here and the experiment in the paper. The version of Trepan we ran here did not use m-of-n splits at nodes (the search for splits becomes very slow if we enable it), we used a different number for the minimum samples required at each node (1000), and we used a simpler rejection-based sampling scheme for generating synthetic samples. The details of the neural network learning algorithm are also different.
This notebook is available [in a gist](https://gist.github.com/sverchkov/c87b301db1b88e0f4cc8bb7d77b889b9).
For issues/questions about the `generalizedtrees` package contact us through https://github.com/Craven-Biostat-Lab/generalizedtrees