Martin Thoma A blog about Code, Politics, and my life

Feature Importance

· by

Contents

Trust is important for a Data Scientist. If you are in a position where you can apply a classification / regression model where the company used rules before, you have to be able to build trust that your model is better than the old system. Stakeholders want to understand what the model does. If you have a rule-based system in your mind, a very natural question is:

What is the most important feature?

This question is problematic and having a feature importance example for trees in sklearn without a fat warning doesn't help. In the past, I failed to give a simple, conclusive answer to why this is problematic. This article fixes this issue.

Feature Selection

Before I come to the examples, I want to introduce two simple ways of feature selection. Feature selection is the process of selecting which features you want to use for your models. The intention of having a small feature set is to keep the model as simple as possible. Simpler models tend to be more robust and easier to understand. In the simplest case, this is done manually. But there are also two intuitive ways to do it automatically: A constructive approach and a destructive approach.

Both are greedy algorithms.

You can find more feature selection algorithms in sklearn.

Feature Construction

Let's say you have \(n\) features in total and a model of type \(m\) (e.g. a Decision Tree). Then you train \(n\) instances of \(m\), each of them on a different feature. You select the feature of the best model and repeat the process. Stop once your improvement is below a threshold.

In pythonic pseudocode:

def construct_features(feature_list, BaseModel, threshold, score):
    """
    Constructive feature selection algorithm

    Parameters
    ----------
    feature_list : list
    base_model : class
    threshold : float
    score : function
        Takes model as input and returns a float between 0 and infty, where
        higher is better.

    Returns
    -------
    features : list
    """
    trained_models = []
    features = []  # Set of current best features
    nb_features = 1
    condition = True
    last_score = 0
    while condition:
        for feature in feature_list:
            m = BaseModel(nb_features)
            m.train(features + [feature])
            trained_models.append((score(m), m, feature))
        features.append(sorted(trained_models)[0])

        # Check if it is time to stop
        condition = score(m) - last_score > threshold
        last_score = score(m)
    return features

Feature Elimination

The same as construction, but the other way around: You start with the complete feature set and remove one feature at a time, trying to keep the score as high as possible.

XOR-Example

2 Features

Let's start simple: We have 2 binary features and one target:

x1  x2    y
------------
0   0     0
0   1     1
1   0     1
1   1     0

Now, what is the most important feature in this case? x1 or x2?

You can easily see that having only one feature cannot be better than random in this case! You need both features and a non-linear classifier to get better than random!

3 Features

A similar setting as before: 3 binary features and the target is XOR of all of them:

x1  x2  x3    y
---------------
0   0   0     0
0   0   1     1
0   1   0     1
0   1   1     0
1   0   0     1
1   0   1     0
1   1   0     0
1   1   1     1

Things to note here:

  • If you take only one feature, then the target \(y\) is exactly 50% of the time equal to that feature and exactly 50% of the time the opposite. No matter which feature you take.
  • If you take two features, you don't get any more information as the target then completely depends on the last one.

From those observations, you can conclude that neither one nor two features can be better than random at predicting the target. No matter which type of model you take. It's simply impossible. The important information is there, but only in combination.

n Features

The principle shown in the two examples above generalizes. If your target is XOR of the features, then you need all features to be better than random!

Soft-XOR

You might argue that this is an artificial example. I agree.

So let's make it less artificial. Take the features and labels from the 3-XOR example. It is essentially a \(8 \times 4\) matrix. Just add a random value in \([-0.01, 0.01]\) to each value. The problem itself didn't change, but it looks less artificial.

Now, you might argue that XOR-like feature behaviour is artificial. That might be true as well, but especially business data could also simply count as artificial.

What are alternatives?

Now that I've explained why feature importance is misleading and needs to be taken with great caution, what are alternatives to explain what the model does?

  • Model lab: See here. Letting stakeholders poke the model.
  • lime: Local Interpretable Model-Agnostic Explanations
  • Common Language: Use the same language as your stakeholders / the community to explain what your model does. Especially when it comes to the metrics you use.

When can we use feature importance?

If the problem is linear / has a big linear part. So if a linear model gives good results, then the feature importance makes sense. A linear model is of the form

$$y := \sum_{i=1}^n a_i \cdot x_i$$

So each feature \(x_i \in \mathbb{R}\) has a weight \(a_i \in \mathbb{R}\). This weight is the importance of the feature.

How do libraries calculate feature importances?

sklearn trees and forests have the attribute feature_importances_. It is the mean decrease in impurity (MDI): every time a node splits on a feature, the impurity (Gini, log loss or mean squared error) of the children is lower than the impurity of the parent. This decrease, weighted by the share of training samples that reach the node, is added to the feature. For forests it is averaged over all trees, and the values are normalized so that they sum up to 1. The sklearn documentation warns that MDI is computed on the training set and favors features with many distinct values.

The alternative in sklearn is permutation_importance: shuffle one feature column of a validation set and measure how much the score drops. It works for any model. For the 3-XOR example with a perfect model, shuffling any single feature drops the accuracy to 50%. So all three features get the same high importance, although none of them is useful on its own. Permutation importance tells you what the model relies on, not which feature you could use alone.

CatBoost has several importance types:

  • PredictionValuesChange (default for most losses): how much the prediction changes on average if the feature value changes. Normalized to sum up to 100.
  • LossFunctionChange (default for ranking): the difference between the loss of the model with and without the feature. It depends on the dataset and can be negative.
  • ShapValues: the contribution of each feature to a single prediction.

TL;DR

Feature importances don't make any sense if the problem is non-linear enough. Many problems, however, could potentially be described in a linear way.