Every device went Warning at once

Symptom

All (or nearly all) devices flip to Warning at the same moment. SNMP data stops everywhere, dashboards freeze on stale numbers, maybe a “no live agents available” style message appears. Individual devices look “broken” but nothing on the network actually changed.

The rule

A simultaneous fleet-wide change is never the devices. Hundreds of independent OLTs and switches do not fail in the same second; the one thing they share is the path through the NetSense agent. Don’t chase individual devices; walk this ladder instead.

1. Is the agent VM running?

The most common cause by far, and half the time the outage is intentional: infrastructure work, a host migration or a planned shutdown that nobody connected to monitoring. Check that the agent VM is powered on and reachable, and ask around whether someone in your team stopped it deliberately.

A stopped agent silently stalls the entire deployment, and it can go unnoticed for days because nothing “errors”, data just stops. NetSense monitors the agent itself and we proactively reach out when we see it down, but if you notice first, checking the VM is step one.

2. Was the VM recently rebooted?

After a VM reboot the agent normally starts automatically and reconnects within a minute or two. If devices are still Warning several minutes after a reboot, tell support the VM was restarted: in some situations the agent service needs a kick from our side, and it’s a one-minute fix once we know.

3. Certificate or DNS problems on the agent

Two platform-level causes produce exactly this picture:

• TLS certificate renewal failed. The agent’s certificate renews automatically every ~3 months via Let’s Encrypt, which needs ports 80/443 reachable and working DNS. If your firewall closed those ports since setup, the next renewal fails and the agent’s backend connection eventually breaks. If your security policy keeps 80/443 closed, coordinate a renewal window with us.
• The agent’s DNS resolver broke. With no working DNS the agent can’t reach the backend, and every device times out at once. If your resolver or its firewall rule changed recently, that’s the suspect.

4. Did NAT or firewall rules change?

The agent needs ports 8080/8081/8082/9090 open to the backend at all times, and behind NAT they must be forwarded 1:1 (same external and internal port). A firewall migration or a “cleanup” of unrecognized rules produces the same everything-at-once symptom. See the install guide’s firewall checklist for the full list.

5. Is the agent VM overloaded?

If polling is slow rather than dead, and devices flap between states or show high latency, check the VM itself: CPU steal on an oversubscribed host, another workload sharing the VM, or an undersized machine. A quick discriminator: latency tests from the agent to LAN device IPs (not to public internet addresses). If LAN latency is high while the network is fine, the VM is the bottleneck, not the network.

What to send support

If the ladder doesn’t resolve it, one message saves a round-trip: whether the VM is up and since when (uptime), whether it was rebooted or migrated recently, and whether any firewall/NAT/DNS change happened around the time data stopped. We can see the agent’s health, availability history and connection state on our side and will usually meet you in the middle.