Regression

Today, I played a bit with the International airline passengers dataset.

It's only feature is a date given by year and month. One should predict the number of passengers (in thousands).

Code

See Github.

Scoring functions

In the following, \(y\) is the ordered list of target, \(y^P\) is the list of predictions in the same order and \bar{y} is the mean of \(y\).

See also:

• What is the difference between “coefficient of determination” and “mean squared error”?

Results

I like the median absolute error best, because it tells me how many passengers my prediction is typically away from the true prediction.

Please note that bad results here do not mean the regressor is bad. Some might just use their full potential with more data, some might be better suited to different regression problems, e.g. interpolation instead of extrapolation or working with higher dimensional data.

Graphs

Plotting the prediction is crucial for extrapolation. What we can see in those images is that there are two things to match: A global trend and local seasonality-effects. To make it easier to digest, I already put them in the table. Just click on them to see them in full size:

Model	Trend	Seasonality	Graph	Comment
RBF SVR	✘	✘		Doesn't learn anything useful at all
GaussianProcessRegressor	✘	Kind of		Overshoots extremely
AdaBoostRegressor	✘	✔
BaggingRegressor	✘	✔
XGBoost	✘	✔
ExtraTreesRegressor	✘	✔		Note the extreme overfitting
RandomForestRegressor	✘	✔
GradientBoostingRegressor	✘	✔
Lasso	✔	✘
Linear SVR	✔	✘
RANSACRegressor	✔	✘
LinearRegression	✔	✘
ElasticNet	✔	✘
SGDRegressor	✔	✘
HuberRegressor	✔	✔		Both are captured pretty bad
PassiveAggressiveRegressor	✔	✔		Both are captured pretty bad

Seeing this result, I decided to have two models: One which predicts the trend and one which predicts seasonality effects.

Combining models

The simplest way to combine two regression models is by

But I also want to simpify the training. So I decided to have the following training setup:

1. Fit \(f_1\) on the training dataset, where \(f_1\) has to predict the global trend and thus solve the extrapolation problem. Lasso seems to be a good choice.
2. Transform the training data \((X, y)\):
   $$y' = y - f_1(X)$$
3. Fit \(f_2\) on \((X, y')\). So \(f_2\) has to solve an interpolation problem.

I call this model TrendSeasonRegressor. Let's try a few combinations:

$f_1$	$f_2$	MAE	MAD	$R^2$	explained variance	MSE	Image
GradientBoostingRegressor		40.0	31.0	0.5689	0.6246	2631.9
Linear Regression	ExtraTrees	19.1	14.5	0.8958	0.9096	636.1
Lasso	ExtraTrees	20.8	16.6	0.8803	0.8992	730.9
ElasticNet	ExtraTrees	25.2	24.4	0.8382	0.8403	988.0
LinearSVR	ExtraTrees	36.4	31.1	0.6841	0.7081	1928.2
LinearSVR	Adaboost	33.7	32.6	0.6918	0.7849	1881.5
Lasso	Adaboost	26.5	18.9	0.8034	0.8143	1200.4
ElasticNet	Adaboost	26.5	19.4	0.8098	0.8229	1161.2

This looks a lot better! Note how the best model before is now the worst!

However, even with this combination the Gaussian Processes were really bad. Overshooting like hell.

Next thing to improve: The scaling of the the seasonal trend is not captured so far.

Another idea I want to dig into is that of residual learning. Where I have only one model learning the residual of another model, you can chain arbitary many. The ResidualRegressor is a meta-regressor. If it is given one model, it behaves exactly like that model. If it is given two models, then it behaves exactly like the TrendSeasonRegressor.

Sadly, for the given data and the models I've tried, three models behaved not better than two in the ResidualRegressor.

Human Performance

An interesting excercise is to get a human baseline. This gives you some insights in how good the models are.

absolute_errors = [15, 36, 11, 30, 23, 15, 23, 4, 31, 10, 28,22,11,27,26,13,5,7,4,41,11,38,35,8,24,42,19,20,8]

Circular Features

Having the month as a single feature with values from 1 to 12 seems not to be the best choice. Either a one-hot encoding or a circular encoding (two features with the trigonometric functions sin and cos) seem to be the better choice.

Using the circular encoding was interesting. It made the prediction to get "bumps":

However, both encodings did not yield better results than the simple integer encoding of a month for this dataset and all tried regressors.

TODOs

• Make a hand-crafted model
• Get a human baseline
• Make a neural network model

Learnings

The following should have become clear from reading this blog post:

• L1: SGDRegressor without scaling is crazy bad.
• L2: Without hyperparameter optimization (e.g. twiddling), SVR is shitty.
• L3: Without scaling, linear SVR runs many hours for training. On a dataset with 3 features and less than 150 data points.
• L4: Visualizations are powerful for model evaluation and improvement in low-dimensional regression problems.

See also

• Comparing Classifiers
• Sklearn: Choosing the right estimator