Backtests

Do the models beat the baseline?

Each model predicts a moto it has never seen, in chronological order, and is scored on the prediction rather than on a fitted value. The design was written down in docs/backtests.md before any result was read.

Pre-registered. Walk-forward by moto, per class, with no feature that sees the moto it predicts. The field average is the baseline every other model has to beat; a model that does not beat it has not earned a place on the site.

Loading…

Overall

MAE is the mean absolute finishing-position error. Brier and log loss are for the win, podium and top-ten probabilities. A lower number is better in every column.

Skill over the field, with intervals

The share by which each model's Brier score beats the field average, bootstrapped over motos (— resamples). An interval that spans zero means the model has not shown a real edge on this sample.

By season

The best model in each season by MAE, and how many motos it was scored on.

By class

The same comparison inside each class, which is the unit the ratings are built on.

Full method, calibration bins and the pre-registration text: docs/backtests.md.