Availability belongs to the service, not the component
A user experiences a service: an application that responds, a transaction that completes. That service depends on a chain of layers, and it is available only when every layer it needs is available at the same moment.
For layers in series, availability multiplies. As an illustration only, three layers that are each available 99.9% of the time give a combined figure of about 99.7%. Redundancy raises the figure for a layer by placing two or more units in parallel, so that the layer fails only when all of them fail together. The design task is therefore to find which layer limits the whole chain, not to buy the most resilient version of every component.
Walk the stack, layer by layer
The table lists the nine layers that should each be examined. For every layer, the question is the same: what happens to the service if one unit here fails, and what happens during planned maintenance?
| Layer | Typical failure | Common mechanisms | What to verify |
|---|---|---|---|
| Power | Feed loss, UPS fault, distribution fault | Dual feeds, A/B distribution, UPS, generator | Each device with two power supplies is wired to both feeds |
| Compute | Host or component failure | Clustering, host restart, spare capacity | Remaining hosts can carry the load after one fails |
| Storage | Controller, shelf or path failure | Dual controllers, redundant paths, replication | Multipath is active and tested, not only configured |
| Network | Switch, link or route failure | Redundant uplinks, dual core, routing convergence | Convergence time is within application tolerance |
| Virtualisation | Host failure, management loss | Cluster HA, live migration, reservations | Running workloads survive loss of the management layer |
| Databases | Instance failure, replica lag | Clustering or replication with defined failover | Failover completes without data loss beyond the stated objective |
| Applications | Process failure, dependency failure | Multiple instances behind load balancers | Session handling survives the loss of an instance |
| Backup | Backup target loss, corrupt backups | Replicated copies, immutable copies | A restore has actually been performed |
| Management systems | Monitoring, identity or DNS unavailable | Redundant management services, out-of-band access | Operators can still reach and manage equipment during a fault |
Remove, tolerate or manage each single point of failure
Every single point of failure found in the review needs a recorded decision. Removing all of them is rarely proportionate, but leaving one unrecorded is a design gap.
- Remove it: add a parallel unit or path, when the cost is justified by the impact of failure.
- Tolerate it: accept the outage, document the expected restoration time and make sure the business owner has agreed.
- Manage it operationally: keep a spare, a runbook and a monitored alert so that restoration is fast and rehearsed.
Look for shared dependencies that defeat redundancy
Two redundant units that share something are not independent. Common examples are two switches fed from one power distribution unit, two firewalls managed from one controller, two sites relying on a single DNS or time source, or two clusters using the same identity service for administrator login.
A dependency walk, starting from each critical application and following every call, name lookup, authentication request and storage path, is the practical way to find these.
Plan for degraded operation
After a failure, the system runs with less capacity. A cluster sized to carry the peak load with all hosts will not carry it with one host down and another in maintenance. Capacity is therefore sized for the load the service must carry during the degraded conditions it is expected to tolerate, which is the same discipline described in the capacity-planning work for the data centre.
Test failover, and record the result
Redundancy that has never been exercised is an assumption. Failover tests should be part of acceptance: remove a power feed, fail a controller, disconnect an uplink, stop an application instance, and record how long the service took to recover and what users experienced. Repeat selected tests after major changes.
- Each layer has a documented failure mode and mechanism.
- Every single point of failure has a recorded remove, tolerate or manage decision.
- Shared dependencies have been traced and documented.
- Capacity has been checked for degraded operation.
- Failover tests have been run with recorded results.
- Monitoring detects each failure and alerts a named owner.