A bus prediction that beats the timetable by four minutes, and why that is not enough
Sixty days of arrival data, a model that works, and the reason I would not ship it.
I logged actual arrivals at the stop by campus for sixty days. The published timetable is wrong by a median of 6.2 minutes. My model is wrong by a median of 2.1 minutes.
That looks like a win and I am not going to ship it.
The reason
The errors are not symmetric and the costs are not either. Predicting the bus comes later than it does means you miss it. Predicting it comes earlier means you wait.
My model is late-biased: it under-predicts arrival time 61% of the time. So the median error improved while the thing people actually care about — missing the bus — got worse than the timetable, which at least errs in the direction everyone already compensates for.
| Timetable | My model | |
|---|---|---|
| Median absolute error | 6.2 min | 2.1 min |
| Share of predictions that would make you miss it | 18% | 29% |
What I am doing about it
Training against an asymmetric loss, where being late costs about four times being early. The median error will get worse. That is the correct trade and it took me two months to see it.
Optimising the number you can compute instead of the one you care about is the easiest mistake there is to make, and it looks like progress the whole way.