Most five-hour fires start as five-minute oddities. A queue depth that ticked up. An error rate that doubled from 0.01% to 0.02%. A background job that finished, but took three times longer than yesterday.
Nobody notices because nobody is looking — not because the team is lazy, but because "looking" was never defined as a task. It sits in the gap between shipping features and getting paged at 2 a.m.
This post makes the case for a daily five-minute ops review. Not a meeting. Not a dashboard you glance at while eating lunch. A deliberate, structured check-in that treats operational health the way a pilot treats a preflight checklist: boring when everything is fine, lifesaving when it isn't.
You might already have alerting. Good. Alerts are necessary. They are also insufficient.
Alerts fire on thresholds. Thresholds assume you know what "bad" looks like in advance. But the failures that burn five hours are the ones that creep. They stay below every threshold until three things compound at once and the system falls over in a way no single alert anticipated.
A daily review catches the creep. You are not looking for red. You are looking for different. Different from yesterday. Different from last week. The human eye is surprisingly good at this — if you give it the same formatted view every day.
Five minutes is the right duration because it is short enough to actually happen. A thirty-minute review becomes a meeting. A meeting becomes optional. Optional becomes skipped. Skipped becomes "we should really start doing that again." You know this story.
The review covers four areas, in order. Order matters because the first area is the one most likely to affect real users right now.
1. Errors and failures. Not every error — just the outward-facing ones. API responses in the 5xx range. Failed webhook deliveries. Timed-out requests. You want the count and the trend. A count of twelve means nothing without knowing yesterday was two.
2. Latency. Look at the 95th percentile, not the average. Averages hide the pain. If your p95 response time jumped from 200ms to 800ms, a subset of your users is having a bad day and none of them will tell you. They will just leave.
3. Queue and job health. Background work that is piling up or silently failing. The question: is work being produced faster than it is being consumed? If yes, you have a clock ticking.
4. Resource headroom. Storage, memory, compute — whatever your system leans on. You are not looking for a crisis. You are looking for a trend line that, extended two weeks, becomes one.
Four areas, roughly a minute each, plus a minute to note anything worth investigating later.
Equally important. During the five-minute review, you ignore:
The discipline of ignoring is what keeps the review at five minutes.
At ten active users, errors are easy to spot because you probably know each user by name. Your review might literally be: "Did anything break? No? Good."
At fifty users, patterns start mattering more than individual events. You need actual numbers, not gut feel. This is when the four-area structure pays for itself.
At two hundred users, you will likely split the review by surface area — one person checks the API layer while another checks background jobs. The format stays the same. The scope per person narrows.
At five hundred-plus, the five-minute review becomes a triage layer. Its job is not to find every problem but to decide which problems deserve a deeper look today. A filter, not a telescope.
The checklist adapts. The habit does not.
Stripped to its core. Adapt the specifics to your system.
Daily Ops Review — 5 minutes
No action is required most days. That is the point. The value is in the days when line three says "yes" and you catch it twelve hours before it would have woken you up.
Building the checklist takes ten minutes. Doing the review takes five. Doing it every single day — that is the hard part.
Tie it to something you already do. First thing after opening your laptop. Right after standup. Immediately before lunch. The trigger matters less than the consistency.
After two weeks, it stops feeling like a chore. After a month, skipping it feels wrong — the way leaving the house without checking the stove feels wrong.
Five minutes a day. That is the price of not being the person who spends five hours on a Saturday rebuilding something that gave plenty of warning.
Be the first to comment.
0 comments
Loading comments...