Forecasting

Today, I played a bit with the International airline passengers dataset.

Its only feature is a date given by year and month. One should predict the number of passengers (in thousands).

Code

See GitHub.

Results

The scoring functions are explained in the regression article.

I like the median absolute error best, because it tells me how many passengers my prediction is typically away from the true value.

Please note that bad results here do not mean the regressor is bad. Some might just use their full potential with more data, some might be better suited to different regression problems, e.g. interpolation instead of extrapolation or working with higher dimensional data.

Graphs

Plotting the prediction is crucial for extrapolation. What we can see in those images is that there are two things to match: A global trend and local seasonality effects. To make it easier to digest, I already put them in the table. Just click on them to see them in full size:

Model	Trend	Seasonality	Graph	Comment
RBF SVR	✘	✘		Doesn't learn anything useful at all
GaussianProcessRegressor	✘	Kind of		Overshoots extremely
AdaBoostRegressor	✘	✔
BaggingRegressor	✘	✔
XGBoost	✘	✔
ExtraTreesRegressor	✘	✔		Note the extreme overfitting
RandomForestRegressor	✘	✔
GradientBoostingRegressor	✘	✔
Lasso	✔	✘
Linear SVR	✔	✘
RANSACRegressor	✔	✘
LinearRegression	✔	✘
ElasticNet	✔	✘
SGDRegressor	✔	✘
HuberRegressor	✔	✔		Both are captured pretty bad
PassiveAggressiveRegressor	✔	✔		Both are captured pretty bad

Seeing this result, I decided to have two models: One which predicts the trend and one which predicts seasonality effects.

Combining models

The simplest way to combine two regression models is by

$$f(x) = f_1(x) + f_2(x)$$

But I also want to simplify the training. So I decided to have the following training setup:

1. Fit $f_1$ on the training dataset, where $f_1$ has to predict the global trend and thus solve the extrapolation problem. Lasso seems to be a good choice.
2. Transform the training data $(X, y)$: $$y' = y - f_1(X)$$
3. Fit $f_2$ on $(X, y')$. So $f_2$ has to solve an interpolation problem.

I call this model TrendSeasonRegressor. Let's try a few combinations:

$f_1$	$f_2$	MAE	MAD	$R^2$	explained variance	MSE	Image
GradientBoostingRegressor		40.0	31.0	0.5689	0.6246	2631.9
Linear Regression	ExtraTrees	19.1	14.5	0.8958	0.9096	636.1
Lasso	ExtraTrees	20.8	16.6	0.8803	0.8992	730.9
ElasticNet	ExtraTrees	25.2	24.4	0.8382	0.8403	988.0
LinearSVR	ExtraTrees	36.4	31.1	0.6841	0.7081	1928.2
LinearSVR	Adaboost	33.7	32.6	0.6918	0.7849	1881.5
Lasso	Adaboost	26.5	18.9	0.8034	0.8143	1200.4
ElasticNet	Adaboost	26.5	19.4	0.8098	0.8229	1161.2

This looks a lot better! Note how the best model before is now the worst!

However, even with this combination the Gaussian Processes were really bad. Overshooting like hell.

Next thing to improve: The scaling of the seasonal trend is not captured so far.

Another idea I want to dig into is that of residual learning. Where I have only one model learning the residual of another model, you can chain arbitrarily many. The ResidualRegressor is a meta-regressor. If it is given one model, it behaves exactly like that model. If it is given two models, then it behaves exactly like the TrendSeasonRegressor.

Sadly, for the given data and the models I've tried, three models did not perform better than two in the ResidualRegressor.

Human Performance

An interesting exercise is to get a human baseline. This gives you some insights into how good the models are.

absolute_errors = [15, 36, 11, 30, 23, 15, 23, 4, 31, 10, 28,22,11,27,26,13,5,7,4,41,11,38,35,8,24,42,19,20,8]

Circular Features

Having the month as a single feature with values from 1 to 12 seems not to be the best choice. Either a one-hot encoding or a circular encoding (two features with the trigonometric functions sin and cos) seem to be the better choice.

Using the circular encoding was interesting. It made the prediction get "bumps":

However, both encodings did not yield better results than the simple integer encoding of a month for this dataset and all tried regressors.

TODOs

• Make a hand-crafted model
• Get a human baseline
• Make a neural network model

Learnings

The following should have become clear from reading this blog post:

• L1: SGDRegressor without scaling is crazy bad.
• L2: Without hyperparameter optimization (e.g. twiddling), SVR is shitty.
• L3: Without scaling, linear SVR runs many hours for training. On a dataset with 3 features and fewer than 150 data points.
• L4: Visualizations are powerful for model evaluation and improvement in low-dimensional regression problems.

See also

• Comparing Classifiers
• Sklearn: Choosing the right estimator