Ask a room of engineers which tasks they dread inheriting from a colleague and the answers come fast: “the restore thing”, “that certificate job”, “whatever Sanne does on the first of the month”. Every team has them, those jobs that work beautifully until the one person who understands them is out of reach. Relief is closer than it looks, because documenting every duty is not required. Start with the right nine DevOps runbooks and you will cover most of the pain. Here is the list, in rough order of how soon we would write them.
For each item you will find why it earns its place and what the page should contain. Keep each runbook short: one task, one page, numbered steps.
1. The service restart
It sounds trivial, and that is exactly why it goes undocumented. A restart that is “just restart it” for the person who built the service may involve draining connections, checking dependent services and confirming health afterwards. Write the sequence down, including the check that proves the service is genuinely back, not merely running.
2. The rollback
If you only write one page this month, make it this one. A rollback page names the previous good version, shows where it is stored, lists the exact commands and states the point beyond which going back is no longer simple, for instance after a one-way database change. Put it at the top of every release checklist.
3. The standard release
Even with a fully automated pipeline, a human usually performs a few steps around it: announcing the release, watching a graph, flipping a switch afterwards. List them, in order, with the person responsible for each. The first time someone else performs a release from your page, you will learn which steps were still living in your head.
4. The backup restore
Everybody has backups; far fewer people have a rehearsed restore. Write the procedure for recovering one table or one file, and the separate procedure for recovering a whole system. Test both on a scratch environment and record how long they took. Knowing the real duration is valuable when someone asks “how long will this take?” in the middle of an incident.
5. The certificate and secret rotation
Expiring certificates are a classic cause of sudden, embarrassing outages. A good page lists every place a certificate is installed, how to request a new one, the order in which to replace them and how to verify each. Reference the vault where secrets live rather than copying them anywhere.
6. The access request
Granting and removing access is dull, repetitive and risky when done from memory. The page should say who approves what, where requests are tracked, how access is granted and, importantly, how it is withdrawn when someone leaves a team. Most teams find the removal half is the one nobody ever wrote down.
7. The new-joiner setup
Every new engineer needs the same accounts, repositories, tools and introductions. A checklist-style runbook turns a chaotic first week into a calm one and frees the team from answering the same questions repeatedly. Ask each new joiner to fix any step that was wrong; they are the best auditors of this page. If you are mapping your own growth, a piece such as this walk through the road from junior to senior engineer shows how this kind of ownership of shared routines is often what moves people up.
8. The on-call handover
When the shift ends, whoever is leaving should not depend on a rushed conversation. A short template covers open issues, anything that looks fragile, temporary changes that must be undone and who is reachable as backup. The template is the runbook; the engineer only fills in the live details.
9. The escalation page
Finally, a map of who to call and when. It lists the first responder, the backup, the manager to inform, and the external contacts, such as a cloud provider’s support channel, with clear criteria for each level. Under stress, people forget even obvious contacts, so write them down and check the list every quarter.
How to write all nine without losing a month
Nine pages sounds like a lot, but each takes a couple of hours when approached properly. The method is the same every time:
- Choose one task and have the person who knows it do it while someone else takes notes.
- Turn the notes into numbered steps that begin with verbs.
- Add what success looks like and when to stop.
- Have a third person follow the page cold, in a safe environment.
- Fix every spot where they hesitated.
- Date it, name an owner and file it where everyone can find it.
Do one per week and you will finish before the quarter ends. Treat them as shared property: any engineer may edit a page the moment they find it wrong.
Not sure which one to start with?
Use a quick scoring habit. For every candidate task, give it a number from one to five for each of two questions. How bad is it if this goes wrong? And how likely is it that someone other than the usual person will have to do it in the next six months? Multiply the two numbers and start with the highest score. Teams are often surprised that the humble access request outranks the glamorous rollback, simply because it happens so much more often.
It also helps to listen to the grumbling. The task people sigh about, the one that always seems to land on the same colleague, is almost always a good first candidate. Sighing is a signal that the knowledge is concentrated and the job is repeated, which is precisely the profile of a task that needs a page.
Finally, remember that a rough page beats a perfect plan. Publish the first version quickly, mark it “draft, tested once”, and improve it each time somebody uses it. Momentum matters more than polish in the first month. A small library that people trust will always do more good than a grand documentation project that stalls halfway through.
The mistakes that quietly ruin runbooks
- Too much prose. If a page reads like an essay, nobody will use it at 3 a.m.
- Screenshots without text. They go out of date the moment an interface changes, and they cannot be searched.
- Missing owners. A page nobody owns stops being true within months.
- Copy-pasted secrets. Never. Point to the vault.
- No test. A page that nobody else has followed is a hope, not a procedure.
Where this fits into your growth
Writing runbooks is not only good for the team. It is a skill employers notice. A roadmap covering the skills, projects and growth of a DevOps engineer is more convincing when your portfolio includes procedures other people have actually used. Being able to say “my restore page was followed by a colleague during a real outage” is worth more than a long list of tools.
Borrowing from your neighbours
You will find plenty of inspiration beyond your own stack. One handy browse for anyone who works near data is TheDataOps.org, whose articles cover how data teams try to keep pipelines dependable, and nearly every idea there has a cousin in the nine pages above.
Quick recap: your nine, in order
- Service restart
- Rollback
- Standard release
- Backup restore
- Certificate and secret rotation
- Access request
- New-joiner setup
- On-call handover
- Escalation page
Print the list, pin it near the team board and cross items off as pages are tested, not merely written. A month from now, the next time somebody is off sick, you will notice something delightful: nothing at all happens. No panicked messages, no waiting for someone’s return. That quiet is the whole point of the exercise, and it is entirely within reach.