Taming alarm noise on a new deployment

Every fresh deployment starts noisy. The defaults are deliberately sensitive (you’d rather see too much on day one than miss a real fault), and your network has quirks the defaults can’t know: ports that are legitimately empty, ONUs that run hot because your climate runs hot. The goal of the first week is simple: get to the point where an alarm means “act now”. At one large deployment, a single over-broad utilization alarm once fired about 15,000 times a day; nobody acts on that, so effectively nothing is monitored.

Here’s the cleanup, in the order that removes the most noise fastest.

1. Suppress what you know is idle

Newly added switches will immediately raise port-down alarms for every unused port, which is correct behavior pointed at the wrong problem. Select the alarms from ports you know are intentionally empty and use Suppress → Until OK: the alarm stays silent until the port actually comes up, at which point monitoring resumes automatically. This is the single biggest noise reduction on a new deployment and takes minutes.

The same applies during planned maintenance: suppress the affected devices’ alarms for the window instead of learning to ignore them.

Note

A manually shut (admin-down) port does not raise the normal port-down alarm at all, there is a separate “PON Port Shutdown” alarm type for detecting admin-down PON ports. So you don’t need to suppress anything for ports you’ve deliberately shut.

2. Fit thresholds to your plant and climate

Defaults are tuned for temperate, air-conditioned plants. Two adjustments come up on almost every deployment in a hot climate:

• PON/ONU high temperature: the default threshold of 65°C is too low where ambient runs hot, many perfectly healthy ONUs sit at 66-67°C in Indian summers. Raising the threshold to ~69-70°C keeps the alarm meaningful. (For reference, ONUs in air-conditioned PoPs typically run 45-50°C.)
• Optical thresholds: instead of one hard-coded dBm number for every port, compare Rx power against the transceiver’s own vendor thresholds, the rule builder supports reading-to-reading comparison, which adapts per optic automatically.

3. Put delays on everything that fluctuates

Packet loss, utilization and CPU all blip. A 3-5 minute alarm delay makes those alarms fire only when the condition persists, and a keep-alive stops a value hovering at the threshold from flapping the alarm open and shut. Keep hard faults (device down, PSU failed) at zero delay: those you want the second they happen.

4. Scope port alarms with filters

Different ports deserve different alarms. Your uplinks are under your control and can page someone at 90% utilization; a customer-facing access port at 90% may just be a good customer. Use match and filter conditions to split them: filter by port description (which is also a reason to maintain descriptions), by hostname convention, or by port type, and give each population its own thresholds and severity. The same technique excludes virtual interfaces (docker, bridges, tunnels) that are “down” by design.

5. Keep severities honest

Reserve Critical for service-affecting faults that demand immediate action. The moment routine events arrive as Critical, your team stops treating Critical as urgent, and you’ve lost the most valuable signal you have. Major for “will affect service if ignored”, Minor and Warning for the rest.

The test that tells you you’re done

Open the alarm list at a random moment: if most active alarms are things someone should actually do something about, you’re there. If the list is dominated by known-idle ports, hot-climate ONUs and flapping thresholds, work back through steps 1-4. Ten minutes of tuning per week in the first month pays for itself every night shift after.