Techniques for model analysis:

Prediction-Based:
* Decision boundaries
* LIME
* Feature importance
* SHAP values
* Partial Dependence Plots
* Sensitivity analysis / perturbation importance
* Model parameter analysis
* ELI 5
* Attention mapping / saliency mapping

Error-Based: * Confusion matrix

Data-Based:
* Dimensionality reduction
* Feature correlations

If you're interested in analysis of CNNs, have a look at my masters thesis:

Analysis and Optimization of Convolutional Neural Network Architectures

## Decision boundaries

Drawing this is only an option if you have 3 or less features. So not really useful in most problem settings.

## SHAP values

SHAP Values (an acronym from SHapley Additive exPlanations) go in the direction of feature importance.

Let me explain them with an example of the Titanic dataset: You have a survival probability of a given person, e.g. 76%. You want to understand why it is 76%.

So what you can do is to twiddle the features. How does the survival probability change when the person has less / more siblings? When the person has the median number of siblings?

There is the `shap`

package for
calculating the shap values.