Techniques for model analysis:
Prediction-Based:
- Decision boundaries
- LIME
- Feature importance
- SHAP values
- Partial Dependence Plots
- Sensitivity analysis / perturbation importance
- Model parameter analysis
- ELI 5
- Attention mapping / saliency mapping
Error-Based:
- Confusion matrix
Data-Based:
- Dimensionality reduction
- Feature correlations
If you're interested in the analysis of CNNs, have a look at my master's thesis:
Analysis and Optimization of Convolutional Neural Network Architectures
Decision boundaries
Drawing this is only an option if you have 3 or fewer features. So not really useful in most problem settings.
SHAP values
SHAP Values (an acronym from SHapley Additive exPlanations) go in the direction of feature importance.
Let me explain them with an example of the Titanic dataset: You have a survival probability of a given person, e.g. 76%. You want to understand why it is 76%.
So what you can do is to tweak the features. How does the survival probability change when the person has fewer / more siblings? When the person has the median number of siblings?
There is the shap package for
calculating the SHAP values.