Techniques for model analysis:
Prediction-Based: * Decision boundaries * LIME * Feature importance * SHAP values * Partial Dependence Plots * Sensitivity analysis / perturbation importance * Model parameter analysis * ELI 5 * Attention mapping / saliency mapping
Error-Based: * Confusion matrix
Data-Based: * Dimensionality reduction * Feature correlations
If you're interested in analysis of CNNs, have a look at my masters thesis:
Analysis and Optimization of Convolutional Neural Network Architectures
Decision boundaries
Drawing this is only an option if you have 3 or less features. So not really useful in most problem settings.
SHAP values
SHAP Values (an acronym from SHapley Additive exPlanations) go in the direction of feature importance.
Let me explain them with an example of the Titanic dataset: You have a survival probability of a given person, e.g. 76%. You want to understand why it is 76%.
So what you can do is to twiddle the features. How does the survival probability change when the person has less / more siblings? When the person has the median number of siblings?
There is the shap package for
calculating the shap values.