Starship
Bus

A bus prediction that beats the timetable by four minutes, and why that is not enough

Sixty days of arrival data, a model that works, and the reason I would not ship it.

I logged actual arrivals at the stop by campus for sixty days. The published timetable is wrong by a median of 6.2 minutes. My model is wrong by a median of 2.1 minutes.

That looks like a win and I am not going to ship it.

The reason

The errors are not symmetric and the costs are not either. Predicting the bus comes later than it does means you miss it. Predicting it comes earlier means you wait.

My model is late-biased: it under-predicts arrival time 61% of the time. So the median error improved while the thing people actually care about — missing the bus — got worse than the timetable, which at least errs in the direction everyone already compensates for.

TimetableMy model
Median absolute error6.2 min2.1 min
Share of predictions that would make you miss it18%29%

What I am doing about it

Training against an asymmetric loss, where being late costs about four times being early. The median error will get worse. That is the correct trade and it took me two months to see it.

Optimising the number you can compute instead of the one you care about is the easiest mistake there is to make, and it looks like progress the whole way.

8 reactions

One project a week, in your inbox

What was built, the numbers it produced, and what did not work. No other email, and one click to leave.