A two-person on-call rotation is simple because it has no process. You and your cofounder alternate weeks. When something breaks, you fix it. There is no escalation path because there is nowhere to escalate. There is no runbook because the entire system fits in two heads.
This works. Until it doesn't.
The breaking point is not a number, but it tends to arrive somewhere between engineer three and engineer eight. The failure mode is always the same: the people who built the system carry the weight of on-call while newer teammates lack the context to help. Resentment builds. Sleep debt compounds. Good engineers leave. The on-call rotation—something nobody thought about when it was just two people—becomes the single biggest driver of attrition.
Here is a lightweight framework for growing on-call without letting it rot.
At this size, the rotation is a shared calendar and a phone that rings. That is fine. But two habits are worth starting now, because they are nearly impossible to retrofit later.
Write down what you did. Every incident gets a short note: what broke, what fixed it, how long it took. Not a formal post-mortem. A few sentences in a shared doc. This raw material becomes the foundation for runbooks later. The cost of writing it now is five minutes. The cost of reconstructing it from memory in six months is hours you will not have.
Count the pages. Track how many alerts fire per on-call shift. You do not need a dashboard. A tally in a spreadsheet is enough. The number itself does not matter yet—what matters is that you establish the habit of measuring. You cannot manage alert fatigue if you never measured alert volume.
This is where ad-hoc breaks. Two things change at once: not everyone was present when the system was built, and the blast radius of a missed page is larger because more customers depend on you.
Assign runbook ownership. Every alert that can fire should have a corresponding runbook. The runbook does not need to be perfect. It needs to exist, and it needs an owner. Ownership means one person is responsible for keeping the runbook current—not that only one person can follow it. The goal is that any engineer on rotation can respond to any alert without calling the person who wrote the code.
Introduce a second tier. Primary on-call handles the page. If primary cannot resolve it within a fixed window—thirty minutes is a reasonable starting point—it escalates to secondary. Secondary is someone with deeper system context. This is not a hierarchy of importance. It is a pressure valve that keeps the primary responder from drowning in problems they cannot yet solve.
The temptation at this phase is to buy a full incident management platform. Resist it. A second tier, written runbooks, and a page count give you most of the value at none of the overhead.
Now the rotation can be genuinely distributed. An engineer might go on-call for a system they did not build and have never debugged. Structure either scales here or collapses.
Set an alert fatigue budget. Pick a number of pages per shift that you consider acceptable. Anything above that number is a problem to be solved, not a cost of doing business. When the budget is exceeded, the on-call engineer's next task is not feature work—it is fixing the noisiest alert. This creates a feedback loop: the people who feel the pain have the authority and the time to reduce it.
Rotate runbook reviews. Runbooks decay. The system changes and the doc does not. Schedule a quarterly pass where each runbook owner verifies their doc against reality. If an engineer follows a runbook during an incident and it is wrong, that runbook gets reviewed immediately—not at the next quarterly cycle.
Separate on-call from hero culture. At two people, on-call heroics are survival. At ten, they are a symptom. If the same person always picks up the hardest pages, you have a knowledge-sharing problem, not a staffing problem. Pair newer engineers with experienced ones on shadow shifts. The goal is broad competence, not deep dependence on individuals.
Every layer of process you add to on-call costs flexibility. A two-person rotation can change anything instantly. A ten-person rotation with tiers, runbooks, and budgets moves slower. That is the trade-off, and it is worth making.
The alternative is not speed. The alternative is a system held together by the willingness of a few people to sacrifice their nights and weekends. That resource is finite and non-renewable. When it runs out, it does not degrade gracefully. It quits and takes institutional knowledge with it.
If you are at two or three engineers: start writing incident notes and counting pages. If you are at four or five: pick your noisiest alerts and write runbooks for them. If you are above five: set a fatigue budget and enforce it.
None of these steps require a new tool, a new vendor, or a new process document. They require a decision that on-call health is a thing you manage, not a thing you endure.
The teams that scale on-call well are not the ones with the best tooling. They are the ones that treat the rotation as a system—one that deserves the same care as the product it protects.
Be the first to comment.
0 comments
Loading comments...