Monitoring added after go-live is monitoring without a baseline
If monitoring is configured after commissioning, nobody knows what normal looks like, thresholds are guesses and some components are never added. Defining the monitoring model during architecture, and configuring it before testing, lets the test period double as the baselining period.
Start from an inventory of what must be observed
Derive the list of monitored objects from the architecture and the bill of material: servers, storage, network devices, security systems, cloud resources, applications, databases, interfaces, edge devices and field equipment. For each, decide which conditions matter: availability, utilisation, capacity, errors, temperature, power, queue depth, interface health.
Set thresholds from the design and tune them with data
Initial thresholds come from the design, such as capacity limits, redundancy assumptions and application response targets. During testing, collect actual values and adjust, so that thresholds separate real problems from normal variation.
Engineer every alert
- Relevant: it describes a condition that matters to the service.
- Prioritised: severity reflects impact and urgency.
- Actionable: the recipient knows what to check or do, ideally from a linked runbook.
- Assigned: a named team or role owns the response.
- Escalated: an unacknowledged alert moves up after a defined time.
- Closed with evidence: the ticket records the cause and the fix.
Alerts that no one acts on should be removed or retuned; they train operators to ignore the console.
Collect logs with a common clock
Decide which logs are collected, how long they are kept and who may read them. Synchronise time across all systems, so that events from different sources can be ordered when investigating an incident.
Build dashboards for specific audiences
Operations staff need current status and open alerts. Infrastructure and application teams need performance and capacity detail. Security teams need event and change views. Management and service owners need availability and trend summaries. A single dashboard attempting to serve all of them serves none well.
Test the monitoring itself
- Induce or simulate a fault: stop a service, pull a link, fill a disk in a test location.
- Confirm that the condition is detected within the expected time.
- Confirm that the alert reaches the assigned person or team and creates a ticket.
- Confirm that escalation operates when the alert is not acknowledged.
- Confirm that the monitoring system raises an alert when it stops receiving data from a source.
Hand over a monitoring package
At handover, the operations team should hold the monitored-object list, thresholds with rationale, alert routing and escalation matrix, dashboards, log-source list and the baseline measurements collected during testing.