Worked example library

Runbook Examples: Operations, Incidents & Recovery

Start with the worked examples to see complete reasoning, then use the shorter pattern library for variation. Level guidance and frameworks show how the same task changes as the evidence, audience, or assignment becomes more demanding.

Guide Examples 8 Templates

Before you copy

What to notice in the examples

A strong runbook can be followed safely by its intended operator under realistic conditions. It names when to use it and when not to use it, verifies prerequisites and permissions, makes branching decisions explicit, includes stop or escalation points, defines successful completion, and is tested and version-controlled so stale operational steps do not survive unnoticed.

State trigger/use case, scope, intended operator, prerequisites, permissions, safety/security boundaries, and linked plans or tickets.
Provide ordered actions with observable checkpoints rather than unexplained commands.
Make decision branches, retry limits, stop conditions, escalation, rollback, and alternate paths explicit.
Define validation and completion criteria, monitoring, communication, records, and post-action follow-up.
Record owner, version, dependencies, test date, change triggers, and references to systems or credentials through secure mechanisms.

Worked format lab

See complete reasoning, not just isolated lines

Use these fuller examples to see what changes between a recognizable pattern and a finished piece of writing. The examples are original or explicitly illustrative, so they demonstrate structure without inventing real-world evidence.

Worked example 1Service-health alert runbook

Illustrative operations runbook; commands are intentionally abstract.

Trigger
Use when the approved service-health alert fires for API error rate above the operational threshold. Do not use for confirmed security incidents; follow the security incident process.

Prerequisites
On-call operator access, incident channel, health dashboard, dependency dashboard.

Steps
1. Confirm the alert is current and identify affected region/service. Expected result: scope is known.
2. Check dependency health. If a shared dependency is degraded, open or join the dependency incident and do not restart local components repeatedly.
3. If dependencies are healthy, compare current release and capacity indicators with baseline.
4. Apply only the approved reversible recovery action for the observed condition.
5. Validate error rate, latency, and one synthetic transaction.

Stop/escalate
Escalate after one unsuccessful recovery attempt or if data-integrity indicators are abnormal.

Closure
Record action, result, linked incident, and follow-up item.

Why it works: The runbook defines when not to use it, one recovery attempt, validation, and escalation instead of encouraging uncontrolled retries.

Worked example 2Scheduled maintenance runbook

Illustrative controlled change.

Pre-check
Approved change ID exists; backup/rollback readiness verified; maintenance notice sent; operator has required role.

Execution
1. Capture pre-change health snapshot.
2. Apply the approved configuration change.
3. Verify service restart/health checkpoint.
4. Run acceptance tests.

Decision
If acceptance fails, execute the approved rollback path and re-test. Do not continue to later steps while the gate is failed.

Completion
Monitor through the agreed window, attach test evidence, update the change record, and create an issue if a residual problem remains.

Why it works: The runbook uses gates and rollback instead of a happy-path checklist only.

Prompt → finished structure

See the decisions between the assignment and the final form

These transformations make the hidden planning step visible so the template does not become a fill-in-the-blanks substitute for judgment.

Transformation 1Expert memory → runbook

Starting material: Experienced operator can fix the alert but documentation says only “restart if needed.”

Decisions
Observe the real diagnostic sequence, define trigger, prerequisites, expected results, safe decision branches, retry/stop limits, escalation, rollback, validation, and record requirements; test with another operator.

Result: Finished structure: controlled operational knowledge instead of tribal memory.

Transformation 2Vendor instructions → local runbook

Starting material: Source is a generic vendor article with commands and defaults.

Decisions
Map steps to the actual environment, permissions, names, dependencies, change/incident rules, security boundaries, validation, and rollback; remove commands that are not authorized or necessary.

Result: Finished structure: environment-specific controlled runbook, not copied documentation.

Depth by level

Increase the reasoning, not just the word count

Level	What changes	Quality test
Routine operations	Give one trigger, prerequisites, ordered steps, expected result, simple validation, and escalation.	A second operator should be able to follow it without hidden tribal knowledge.
Incident / maintenance runbook	Add branching decisions, stop conditions, rollback, communications, linked change/incident records, and evidence.	Failure paths should be designed before the runbook is needed under pressure.
Sensitive / high-impact operation	Use least-privilege access references, approvals, peer checks where required, safe testing, controlled commands, audit evidence, and specialist sign-off.	The runbook must match the real environment and authorized operating procedure.

Reusable frameworks

Start from the decisions the format requires

Framework 1

Observation → function
1. What can the viewpoint actually perceive?
2. Which 1–2 details matter now?
3. What do those details change in image, pace, relationship, or action?
4. What interpretation remains uncertain?

Framework 2

Generic → specific revision
Generic line: [x]
Observable evidence: [x]
Context/constraint: [x]
Unnecessary inference removed: [x]
Revised line: [x]

Alert runbook: verify the alert source and service scope, check approved health indicators, distinguish dependency failure from local failure, apply the authorized recovery path, validate, and escalate if the stop condition is reached.

Scheduled maintenance runbook: confirm change approval and backup state, announce maintenance, perform ordered steps, validate service health, roll back if the acceptance check fails, and close the change record.

Database failover runbook: confirm incident commander authorization, verify replica state and data-lag evidence, perform the approved failover action, validate application connectivity, and monitor before declaring success.

Account lockout runbook: verify requester identity using the approved process, check lock reason and security flags, follow the authorized unlock/reset path, and escalate suspicious cases rather than bypassing controls.

Batch-job recovery runbook: identify failed stage, confirm whether rerun is idempotent under the actual design, choose restart point, reconcile output, and avoid duplicate processing.

Certificate-expiry runbook: confirm affected endpoint and ownership, validate replacement source, deploy through approved change path, test chain/hostname, and record renewal ownership.

Customer-impact runbook: define internal response steps and link a separate communication template rather than forcing operators to invent external wording during an incident.

Runbook validation: a quarterly test finds one renamed dashboard and a changed permission requirement; update the references and retest the affected branch.

Turn an example into your own writing

Keep the underlying decision or pattern, then replace the subject, evidence, relationship, constraints, and tone with details that belong to your situation. If your final line still works after swapping only one noun, it may be too close to the example.

What to notice in the examples

See complete reasoning, not just isolated lines

See the decisions between the assignment and the final form

Increase the reasoning, not just the word count

Start from the decisions the format requires

Related writing guides