Techniques for model analysis:

Prediction-Based: * Decision boundaries * LIME * Feature importance * SHAP values * Partial Dependence Plots * Sensitivity analysis / perturbation importance * Model parameter analysis * ELI 5 * Attention mapping / saliency mapping

Error-Based: * Confusion matrix

Data-Based: * Dimensionality reduction * Feature correlations

If you're interested in analysis of CNNs, have a look at my masters thesis:

Analysis and Optimization of Convolutional Neural Network Architectures

## Decision boundaries

Drawing this is only an option if you have 3 or less features. So not really useful in most problem settings.

## SHAP values

SHAP Values (an acronym from SHapley Additive exPlanations) go in the direction of feature importance.

Let me explain them with an example of the Titanic dataset: You have a survival probability of a given person, e.g. 76%. You want to understand why it is 76%.

So what you can do is to twiddle the features. How does the survival probability change when the person has less / more siblings? When the person has the median number of siblings?

There is the `shap`

package for
calculating the shap values.