The problem: sites drift
Each site is installed, repaired and adjusted by different people at different times. Without controls, configuration, firmware and settings diverge, so that two sites that should behave identically do not, and a fault at one site cannot be diagnosed from knowledge of another.
Define configuration items and a baseline
Decide which items are under configuration control: device configurations, firmware and software versions, firewall rules, certificates, monitoring thresholds, and the documents that describe them. Record an approved baseline for each site class, as the reference against which actual configuration is compared.
Use templates with site-specific parameters
Hold the common configuration in a template and the site-specific values, such as addresses, names and identifiers, in a separate parameter set. A change to the common part is made once and applied everywhere, and the differences between sites are visible and limited.
Keep configuration in version control
- Store templates, parameters and exported configurations in a version-controlled repository.
- Record who changed what, when and why, linked to a change record.
- Keep the ability to return to a previous version quickly.
- Restrict write access and require review for changes to shared templates.
Detect drift routinely
Retrieve running configuration from devices on a schedule and compare it with the intended version. Differences are either unrecorded changes to be investigated, or emergency fixes that need to be formalised or reversed. Compare firmware and software versions in the same way.
Back up device configuration
Keep current configuration backups for every device, stored outside the device and the site. Replacement of a failed unit is then a matter of restoring a known configuration, rather than reconstructing it from memory.
Roll out changes in stages
- Test the change in a lab or on a non-critical site.
- Apply to a small group of representative sites and observe.
- Extend in waves, with a stated stop condition.
- Validate each site against defined checks.
- Update documentation and the asset and configuration records.
Keep records aligned
The configuration record, the asset register and the network diagrams describe the same system. Update them in the same change, and reconcile them periodically with what is discovered on the network and observed on site.