Starship
Veille

I built a status page and it lied to me for nine days

A monitor that checks the wrong thing is worse than no monitor at all.

I run three small services for student clubs. I built a status page so I would know before they told me.

It checked that each service returned HTTP 200 on the home page. Every ninety seconds, green across the board.

The nine days

One of the services — a sign-up form for a club — had a broken database connection. The home page still rendered, because it is static. Every submission failed. The page stayed green for nine days while the form silently dropped 34 sign-ups.

Nobody told me because the form appeared to work: it showed a confirmation, then did nothing.

What I changed

The check now does what a user does. It submits a real sign-up with a known test address, reads it back from the database, and deletes it. If any step fails, the service is red.

It is slower, it writes to production, and it has caught two real failures in a month.

A check that cannot fail the way your users fail is decoration.

What I owe the club

I told them. They lost thirty-four people who thought they had signed up and were never contacted. That is not a bug report, it is an apology, and the status page did not prevent it — it caused it, by making me stop checking.

4 reactions

One project a week, in your inbox

What was built, the numbers it produced, and what did not work. No other email, and one click to leave.