Reliability Plan Template
The Production Readiness Checklist asks the questions. This is the document that holds the answers.
Copy it, keep one per service, and keep it where your on-call can reach it during an incident rather than in a wiki nobody remembers. It is deliberately short. A reliability plan that takes a day to read is a reliability plan nobody consults at 3am.
Example values are included in italics to show the intended level of specificity. Replace them.
1. Service identity
| Field | Value |
|---|---|
| Service name | orders-api |
| Business owner | name, team |
| Technical owner | name, team |
| On-call rotation | link to schedule |
| Escalation path | first, second, and decision authority for a stop |
| Harper deployment | Fabric cluster, or self-managed, with cluster identifier |
| Harper version | 5.2.x, per node |
| Plan last reviewed | date |
| Next review due | date |
2. Critical journeys
The user-facing paths that must work. Everything else in this plan is stated per journey, so this table defines the scope of all of it.
| Journey | Description | Databases and tables involved | Downstream dependencies | Owner |
|---|---|---|---|---|
| submit order | POST through the public API | orders.orders, orders.items | pricing, payments | |
| browse catalog | cached read | catalog.products | none |
3. Objectives
| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective | Exclusions |
|---|---|---|---|---|---|
| 99.95% | at the CDN edge | 15 min from detection | 5 min | planned maintenance, third-party outages |
Monthly unavailability budget: 21 min 55 sec at 99.95%
Error budget policy: what changes when the remaining budget falls below a threshold, and who decides
4. Architecture and topology
- Node count and
F, the tolerated simultaneous loss: 5 nodes, F = 1 - Failure domain layout: how nodes are distributed, and the proof that domains are independent
- Traffic layer and routing: what fronts Harper, and how it selects nodes
- Replication scope: which databases, which tables excluded, and whether
systemis in scope - Sharding in use: yes or no, and where
- Storage engine per database: per database
5. Capacity
| Measurement | Value | Conditions | Date measured |
|---|---|---|---|
| Peak required throughput | per journey | ||
| Tested per-node throughput | data volume, query mix, peer synchronizing y/n | ||
| Target utilization | derived from latency at the SLO percentile | ||
Surviving capacity at F | measured in the drain drill, not calculated |
Maintenance policy when a drain would breach F: pause, add capacity, or accept with documented duration
6. Traffic admission
- Liveness probe: endpoint, interval, threshold
- Availability flag:
status-checkdeployed how, and who may set it - Readiness route: path, and the dependencies it checks
- Journey synthetic: what it does, from where, how often
- Drain sequence: link to runbook
- Return sequence, including stability window: link to runbook
- Measured time to zero traffic after drain: value, date
- Measured node re-entry time including convergence: value, date
7. Replication
- Measured convergence time under load, per critical database: value, date
- Measured full synchronization time for a new node: value, date
- Admission gate threshold, and how it compares to convergence: value
- Non-commutative operations and their handling: operation, and the owner, serialization point, or coordinator
- Certificate expiry dates and alert lead time: dates
8. Deployment and change
- Artifact source and reference format: registry or git, pinned version format
- Cohort ladder: one node, internal, 5 percent, 25 percent, full
- Stop thresholds and hold times: stated numerically
- Decision authority for advance, hold, stop, roll back: name or role
- Restart mode used, and how completion is confirmed:
rolling, polling the job - Measured release reversal time: value, date
- Forward repair procedures for data-affecting releases: link
- Change record location: link
9. Observability
- Dashboard location: link
- Alerts configured, with thresholds and destinations: table or link
- Metric export path: Grafana on Fabric, or Prometheus scrape
- Log destination and retention: system, retention period
- Deployment annotations enabled: yes or no
- Triage runbook: link
- Measured time to detection, per drill: value, date
10. Backup and recovery
| Database | Mechanism | Cadence | Retention | Off-host destination | Last verified | Last restore drill | Measured restore time | Measured data loss |
|---|---|---|---|---|---|---|---|---|
- Restore authority: who may execute a restore
- Restore constraints per database: online, or offline only because a component holds it open
systemdatabase restore procedure: offline only, link to runbook- Failure classes assigned to restore versus forward repair: table
11. Scenario responses
| Scenario | Mechanism | Runbook | What proves it works | Last exercised |
|---|---|---|---|---|
| single node loss | ||||
| node return after restart | ||||
| node replacement | ||||
| failure domain loss | ||||
| region loss | ||||
| bad release, no data change | ||||
| bad release that changed data | ||||
| accidental destructive operation | ||||
| storage loss on one node | ||||
| certificate expiry | ||||
| downstream dependency outage |
12. Degraded modes
| Journey | Condition | Declared behavior | Who may invoke | Automatic or manual |
|---|---|---|---|---|
| pricing service unavailable | serve last known price, flag the response | |||
| node behind on replication | remove from rotation | |||
| cluster below capacity budget | shed non-critical traffic |
13. Security and access
- Operator roles and privileges: who has what
super_usercredential inventory and rotation: link, schedule- Operations API network restriction: how enforced
- Certificate inventory and expiry alerts: link
- Deploy credential custody: secret store, rotation method
- Administrative access logging, including host-level: system
14. Open risks
Accepted gaps, each with an owner and a date. This section existing and being honest is worth more than it being empty.
| Risk | Impact if realized | Why accepted | Owner | Review date |
|---|---|---|---|---|
15. Revision history
| Date | Author | Change | Trigger |
|---|---|---|---|
| launch, version upgrade, topology change, incident |
Keeping this current
Most of the numbers in this plan are properties of a Harper version, a data volume, a topology, and a set of components. All four change, so the numbers expire. Re-measure and revise after:
- A Harper version upgrade
- A topology change, including adding or removing nodes
- Significant data growth
- Any incident that produced a surprise
Date every number. An undated measurement in a reliability plan is worse than a blank field, because a blank field prompts someone to go and measure.
Additional Resources
- Production Readiness Checklist for the gate that populates this plan
- Engineering RPO, RTO, and Uptime for how to derive sections 3 and 11
- Backup and Recovery for section 10
- Safe Deployments and Rollback for section 8
- Monitoring and Triage for section 9