We almost killed the habit six months before we needed it.
The routine was simple: every morning, one person on the team spent fifteen minutes reviewing every running system. Not a deep investigation. Not a full audit. Just a walk-through. Are things where they should be? Are the numbers inside normal ranges? Does anything look off, even if it isn't technically broken?
We called it the morning sweep. It had no drama. It produced no dashboards anyone wanted to screenshot. Most days, the person doing it would finish in ten minutes and report back with a single word: clean.
The morning sweep started after a minor incident about a year earlier. A configuration change had drifted in a way that didn't trigger any alerts but slowly degraded performance for a subset of customers. Nobody noticed for three days. The fix was trivial — but the trust damage was not.
We didn't build a new alerting system. We didn't buy anything. We added a line to our daily routine: someone looks at things with their own eyes, every morning, before the day starts.
The rotation was informal. Whoever was on support that day did the sweep. They checked known indicators, compared them against what was normal, and flagged anything worth a second look. It took less time than making coffee.
Six months in, someone asked the reasonable question: why are we still doing this?
The argument against was honest. We had monitoring. We had alerts configured for the scenarios we knew about. The sweep had caught exactly two minor issues in six months, both of which would have triggered alerts within hours anyway. Fifteen minutes per day, multiplied across the team, felt like overhead for a small company trying to move fast.
The conversation was short. One person pushed back with a point that stuck: "The sweep isn't for the problems we've already imagined. It's for the ones we haven't."
We kept it. Barely.
Two months later, on a Wednesday morning, the person doing the sweep noticed something strange. A set of background processes were completing successfully — no errors, no alerts — but their completion times had shifted. Maybe 40 percent slower than the day before. Still well within any threshold that would fire an alert.
She flagged it. We looked closer. What we found was a slow resource exhaustion that, left alone, would have cascaded into a full outage within 48 hours. Not a maybe. A certainty. The kind of failure where customers lose data and you lose sleep for a month.
Because we caught it Wednesday morning, we had time. We diagnosed the root cause that afternoon. We deployed a fix by Thursday. No customer was affected. No data was lost. No one outside the team ever knew.
Then Thursday night, an unrelated provider we depend on had their own outage. Our systems handled the disruption, but the team spent Friday firefighting downstream effects — rerouting traffic, communicating with affected customers, stabilizing things manually.
If we had hit Friday carrying the undetected resource problem from Wednesday, we would have faced two compounding failures at once. I don't know exactly what would have happened. I know it would have been bad. The kind of bad where you send emails that start with "We sincerely apologize."
Instead, we came out of the worst week we'd had in a year with zero customer-facing incidents that were our fault. The Friday fire was hard, but it was one fire. We could handle one fire.
The morning sweep looked useless for months because it was quietly doing its job. The absence of surprises felt like evidence the habit wasn't needed. In reality, the absence of surprises was the whole point.
This is the trap with operational discipline. The better it works, the less visible it becomes. The pressure to cut it grows precisely when it's earning its keep.
After that week, nobody suggested killing the sweep again.
There's a strong temptation, especially for founders, to treat reliability as a purchasing decision. Pick the right vendor. Configure the right alerts. Buy the right insurance policy. Move on to building product.
But tools only catch the failures they were designed to catch. Every system has gaps between what is monitored and what is possible. The morning sweep lives in that gap. It doesn't replace monitoring. It complements it with something monitoring can't provide: a human paying attention with no specific expectation of what they'll find.
The habit costs us about twelve minutes a day. It has no interface. It produces no reports. It is, by every measurable standard, boring.
It saved us during our worst week. Not because it was sophisticated. Because it was there.
If you run a small team and you don't have a version of this — a regular, unglamorous, human check on the state of your systems — start one tomorrow. Make it lightweight enough that no one resents it. Rotate it so everyone builds the instinct. Protect it when someone calls it unnecessary.
The habit that saves you will never look important on the day you decide to keep it.
Be the first to comment.
0 comments
Loading comments...