Incidents: one alert per outage

An outage rarely breaks just one thing. When a switch fails or a site loses power, several devices stop answering within a minute or two, and each can send its own alert. Monitaria puts problems that belong together into one incident and lists the probable cause first. Once an admin turns on alert grouping, the team gets one alert per incident instead of one per sensor.

How problems are grouped

• A problem is a sensor that is Down or Warning. A new problem joins an open incident when it is on a device already in the incident, behind a device in it, or upstream of devices in it, like a router that fails a minute after the printers behind it.
• Behind means: devices at a remote site sit behind that site's router or firewall, a sensor set to depend on another sits behind it, and at remote sites a device sits behind the switch that CDP/LLDP shows it connected to.
• Problems at the same site that start within 5 minutes of the incident's first problem join it too; Group problems at the same site that start within (minutes) sets the window. Anything else starts a new incident. Sites are the same as everywhere else in Monitaria (see the sites guide).
• The probable cause is the topmost device: the one most others in the incident sit behind, then routers and firewalls, then switches, then Down before Warning, then whichever failed first. The incident page says why it was chosen.
• A title names the site, the cause and the scale, for example Warehouse: router RB-WH-RT01 down, 14 devices unreachable.

Turn on one alert per incident

Incident tracking is on by default: Problems › Incidents fills up by itself, and nothing is sent anywhere because of it. Alert grouping stays off until an admin turns it on.

1. In Settings › Alerts, make sure alerts have somewhere to go: a mail server if you send email, and rules that email your team or post to a webhook.
2. Open Settings › Incidents. While alerts still go out one per sensor, admins also see a banner on Problems › Incidents with an Open Settings › Incidents button.
3. Tick Group alerts into incidents, keep the default timings to start with, and choose Save. The change applies straight away, without a restart, and is recorded in the audit log.
4. Look at Where incident alerts go at the bottom of the page. A rule gets an incident's alert when it would have got the alert of at least one problem in it, with the same device types, sites and statuses.

The alerts you get

• Opened: one alert once the incident has settled, so it already names the probable cause and how many devices are affected. The wait is 90 seconds by default; change it with Wait before the first alert (seconds). A problem that clears sooner sends nothing but is still listed.
• Update: straight away when something in it goes Down after an alert that had only warnings. When more problems join, you get an update at most every 15 minutes; change it with Send an update when it grows, at most every (minutes).
• Resolved: one alert, saying how long the problems lasted, once every problem in it has been Up again for 5 minutes; change it with Close it after everything has been back up for (minutes).
• Meanwhile the per-sensor alerts for the incident's problems, and their recoveries, are held back. Webhooks get a text line that Teams and Slack show, plus fields such as kind (incident.opened, incident.updated or incident.resolved), incident_id and title.
• Count warnings as problems is on by default. Turn it off and warnings are left out of incidents; their alerts go out as before. If warning alerts are off in Settings › Alerts, an incident with only warnings sends nothing until something in it goes Down.

Check it works

Open Problems › Incidents. Open incidents come first, with filters for Status, Site and Period and a search box, and each count above the list opens what it counts. Open an incident once its first alert is due: the line under the title says whether it alerts as one incident or one sensor at a time, and the Timeline shows each alert with the rules it went through, or why it wasn't sent. Settings › Alerts › Recent deliveries lists the deliveries too.

Everyday use

• Acknowledge all (operators and admins) acknowledges every current problem in the incident that you can see, with an optional note such as a carrier ticket number. Repeat alerts stop, and each acknowledgement clears itself when its sensor recovers.
• Add a note puts a line in the Timeline that everyone who can see the incident can read.
• Problems in this incident shows every member and why it joined. Filter it by Still a problem, Acknowledged, Back up or Behind the cause.
• A sensor's details show Part of incident #N, which opens the incident. A device's details list its recent incidents.
• Problems › Noisiest sensors lists the sensors that went Down or into Warning most often in the last 7 and 30 days, with a suggestion for each: raise Failures in a row before down, adjust a threshold, raise the timeout, or schedule a maintenance window for the hours problems usually start. Admins can choose Edit in Sensors to make the change.

Good to know

• With alert grouping on, rules set to Repeat while still down and not acknowledged send one reminder per incident, at the rule's interval, instead of one per sensor. Acknowledging the incident stops them.
• A problem that joins after the first alert is announced in the next update, not on its own. Incidents are never merged later, and changing the grouping window doesn't regroup existing ones.
• CDP/LLDP links are only used at remote sites. At the collector's own site, devices behind a failed switch join its incident through the 5-minute window instead.
• An incident alert lists the whole incident, including devices outside a rule's device types or sites.
• Problems in a maintenance window don't start or join incidents. Turning grouping on or off only changes incidents that start afterwards.
• If incident tracking fails, alerts stop being held back and go out one by one as before. A failure here never stops monitoring.
• People limited to some devices see only their part of an incident, and its title and probable cause never name a device outside their view. Incidents stay on the collector and the hub doesn't receive them. Closed incidents are kept for 90 days; change it with Keep closed incidents for (days).