What is a server monitoring alert and why does configuration matter?
A server monitoring alert is a notification triggered when a server metric crosses a set threshold, such as high CPU, memory, or disk usage. It tells your IT team when a resource needs attention.
The tricky part is getting those thresholds right. Set them too low, and you’ll be bombarded with false alarms; set them too high, and real issues may slip through. Thresholds, alert duration, severity, escalation rules, and maintenance windows all play a role in turning alerts into useful signals instead of background noise.
Step 1: Establish a baseline before setting any threshold
- Collect baseline data first: Monitor server performance for at least 4—6 weeks before setting thresholds.
- Capture different usage periods: Include both peak and off-peak hours to understand normal behavior.
- Use P95 utilization: Look at the 95th percentile, rather than averages, to account for normal spikes.
- Set server-specific thresholds: Use the baseline to define warning and critical levels that reflect what is actually normal for each server.
- Avoid alert noise: For example, an 80% CPU threshold on a server that regularly reaches 75% during peak hours will likely generate unnecessary alerts.
Note: For newly onboarded servers without historical data, use conservative starting thresholds while the baseline builds:
- CPU: 70% warning / 90% critical
- Available memory: Below 500 MB warning / below 200 MB critical
- Disk space: 75% warning / 90% critical
- Disk I/O queue depth: Sustained above 2 per disk warning / 4 critical
- Page file usage (Windows): 50% warning / 80% critical
- Swap usage (Linux): Any sustained activity warning / 25%+ critical
These starting points help catch potential problems before enough historical data is available. Once 4—6 weeks of data are collected, revise the thresholds based on the server's actual workload and behavior.
Step 2: Configure two alert tiers: Warning and Critical
Every monitored metric should have two thresholds. A single critical-only threshold removes the intervention window: the time between "approaching a problem" and "problem has occurred."
| Severity tier | What it signals | Who gets notified | Typical response |
|---|---|---|---|
| Warning | Metric approaching problem territory | Monitoring team / helpdesk | Investigate, create ticket |
| Critical | Metric at or beyond impact threshold | On-call engineer | Immediate response required |
The gap between warning and critical gives your team time to investigate and act before an issue affects users. A two-tier alert model creates this buffer, while a single threshold turns every alert into an emergency.
Step 3: Set rearm values to prevent alert storms
A rearm value defines when an active alert should clear. Without this, a metric that repeatedly moves above and below its threshold can generate constant alert-fired and alert-cleared notifications, creating unnecessary noise.
- Create enough separation: Set the rearm value 10—15 percentage points below the warning threshold for CPU and memory. Example: If the CPU warning threshold is 80%, set the rearm value to 65—70%.
- Reduce alert noise: The alert fires once when the threshold is crossed and clears only when the metric returns to a stable level.
- Match the operators: If the alert uses a greater-than (>) condition, the rearm should use less-than-or-equal (≤), and vice versa.
Step 4: Configure consecutive poll counts to filter spikes
A single spike above a threshold doesn’t always mean there’s a real problem. CPU usage, for example, may briefly hit 95% during a scheduled task and quickly return to 40%. Alerting on every spike creates false positives and unnecessary noise.
Consecutive poll count helps filter out these short-lived spikes by requiring a metric to exceed its threshold for a set number of consecutive polls before triggering an alert.
The key question when setting poll counts is: can this metric return to normal on its own before the next check? CPU and network usage can spike and then settle back down, so a single high reading may not indicate a real problem. A stopped service, however, will stay stopped until it is restarted. Disk space is similar; once usage crosses a threshold, it won't drop without some action.
Use a poll count of 1 for conditions that won't correct themselves. For metrics that naturally spike and recover, use 2—5 consecutive polls to filter out short-lived fluctuations, depending on how volatile the workload is.
| Metric | Recommended consecutive polls | Rationale |
|---|---|---|
| CPU utilization | 3—5 | CPU spikes are normal; sustained high CPU is the problem |
| Memory utilization | 2—3 | Memory changes slowly; fewer polls needed to confirm a trend. |
| Disk space used | 1 | Disk space does not spike and recover; any reading above threshold is real |
| Service status (up/down) | 1 | A stopped service is always a problem; no delay needed. |
| Network interface utilization | 3 | Traffic bursts are normal; sustained saturation is the problem. |
| Disk I/O queue depth | 2—3 | Short queue depth spikes are expected under load. |
Setting the consecutive poll count to 1 for every metric can quickly lead to alert fatigue by triggering brief spikes. Setting it too high, however, can delay detection of actual issues. Choose a count that filters short-lived spikes without slowing down response to real problems.
Step 5: Configure escalation policies with time-window triggers
An alert is only useful when someone takes action. Escalation policies define who should respond and when an alert should move to the next level.
- Set clear escalation tiers: Notify the monitoring team when an alert fires, then escalate to a senior engineer if it remains unacknowledged. For example, use 15 minutes for warnings and 5 minutes for critical alerts.
- Use role-based contacts: Assign alerts to roles such as “on-call engineer” instead of specific individuals. This keeps policies flexible as team members change.
- Align with SLAs: Set escalation windows based on the server’s business impact. A critical production server may need a 5-minute escalation, while a non-critical internal server may allow 30 minutes.
- Escalate automatically: If an alert remains unacknowledged, move it to the next responsible person without relying on manual follow-up.
Step 6: Configure maintenance windows for planned downtime
Planned activities such as OS patches, reboots, and application restarts can trigger unnecessary alerts if they aren’t scheduled in advance. Maintenance windows suppress alerts during approved downtime and automatically resume monitoring afterward.
- Schedule planned maintenance: Configure a maintenance window before the activity begins to prevent unnecessary alerts and tickets.
- Add a time buffer: Give the window extra time beyond the expected duration. For example, a 30-minute reboot could use a 45-minute window.
- Protect SLA accuracy: Downtime during a maintenance window is treated as scheduled downtime and excluded from unscheduled downtime calculations.
- Resume automatically: Once the window closes, normal monitoring and alerting continue without manual intervention.
Step 7: Alert only on conditions that require action
Before creating an alert, ask: “If this fires at 3 a.m., does the on-call engineer need to do something?” If the answer is no, it probably shouldn’t trigger a notification.
Common non-actionable alerts to remove:
- Informational metrics: Use them for dashboards, reports, and trend analysis instead of triggering alerts.
- Correlated events: If one switch failure affects 50 servers, alert on the root cause and suppress the related server alerts to avoid flooding the alert queue.
- Expected spikes: If CPU consistently reaches 90% during a scheduled batch job, don’t treat it as an anomaly. Adjust the threshold for that period or configure a maintenance window.
Steps 1—7 cover manual threshold configuration and form the foundation of an effective alerting strategy. AIOps-based adaptive thresholds automate the first two steps by learning each server’s normal behavior and adjusting warning and critical levels as usage patterns change. However, rearm values, consecutive poll counts, escalation policies, and maintenance windows still need to be configured to keep alerts useful and reduce noise. Even with AIOps, understanding manual thresholds remains important: the system can learn what’s normal, but it doesn’t decide who gets alerted or when an alert should clear.
How does OpManager handle alert and threshold configuration?
OpManager gives you flexible alert controls to help reduce noise and catch issues early.
- Flexible thresholds: Configure warning, critical, attention, and trouble levels, along with rearm values and consecutive poll counts. Apply settings to individual monitors, devices, or entire device groups.
- Template-based configuration: Use device templates to apply threshold settings across multiple servers at once. Update a template once, and the changes are automatically applied to all associated devices.
- Adaptive thresholds: OpManager’s AIOps engine learns each server’s normal behavior and dynamically adjusts thresholds. This helps account for workload patterns that change by time of day or week.
- Smart escalation: Set multi-level escalation policies with time-based triggers and role-based contacts. OpManager can also integrate with ITSM platforms such as ServiceNow, Jira, and Freshservice to create tickets automatically.
- Maintenance windows: Suppress alerts for planned maintenance and automatically resume monitoring afterward. Scheduled downtime is also classified separately in availability reports to keep SLA calculations accurate.