High availability with a standby collector

A single collector is a single point of failure: if its host loses power or an upgrade goes wrong, nothing watches the network. High availability pairs two collectors at a site. The primary monitors, while the standby keeps a copy of its settings but doesn't check sensors, send alerts or receive flows and logs, and keeps features that reach out, such as config backup, cloud polling, port monitoring, webhooks, tickets and scheduled reports, switched off. If the primary stops answering, the standby takes over in about 30 seconds and hands back once the primary has been running again for 2 minutes.

Before you start

• A second collector that can reach the same devices. The VM appliance, Docker and the Linux installer all work.
• A license for the second collector, for the same number of devices.
• The same Monitaria version on both, and clocks kept in sync by NTP. The two refuse each other's messages when their clocks are more than 10 minutes apart.
• Each collector able to open the other's web address, for example https://192.0.2.11:8443 for the primary and https://192.0.2.12:8443 for the standby.
• With Docker or the installer, a standby can also start paired from its first start: set MONITARIA_HA_ROLE, MONITARIA_HA_PEER and MONITARIA_HA_SECRET on the container, or run the installer with --ha-role standby, --ha-peer and --ha-secret-file.

1. Pair the two collectors

1. On the primary, open Settings › High availability, tick Turn on high availability and choose Primary under This collector is the.
2. Under The other collector's address, enter the standby's address, for example https://192.0.2.12:8443.
3. Choose Generate next to Shared secret, copy the secret and choose Save. Set up the standby straight away: until it answers, the primary warns that nothing would take over.
4. On the standby, open the same page, tick Turn on high availability and choose Standby. Enter the primary's address (https://192.0.2.11:8443) and paste the same secret.
5. Choose Test connection. It should say the other collector answered: the primary, monitoring.
6. Choose Save and confirm. The standby stops monitoring and replaces its own settings with a copy of the primary's within a heartbeat (10 seconds by default).
7. Add the admin accounts you need on the standby too, because user accounts aren't copied, or use single sign-on.

2. Check it works

On both collectors, Status on the same page should show the other collector as answering, and every page of the standby carries the banner This is the standby collector. To test a failover, stop the primary with sudo systemctl stop monitaria (docker stop monitaria for Docker). Within about 30 seconds the standby shows that it has taken over monitoring and sends an alert. Start the primary again and monitoring is handed back about 2 minutes later.

Receivers and a floating address

Routers and firewalls send NetFlow, syslog and traps to one address, and the pair doesn't move addresses itself. Either point devices at both collectors (most can send to two destinations), or give the active collector a floating address with keepalived. Each collector answers /api/public/ha/active on its own machine with 200 while it is active and 503 while it stands by, which keepalived can track. With the sample below, devices send to 192.0.2.10.

# /etc/keepalived/keepalived.conf on both collectors
# (priority 150 on the primary, 100 on the standby)
vrrp_script monitaria_active {
    script "/usr/bin/curl -fsk https://127.0.0.1:8443/api/public/ha/active"
    interval 5
    fall 2
    rise 2
}
vrrp_instance MONITARIA {
    interface eth0
    virtual_router_id 41
    priority 150
    advert_int 1
    virtual_ipaddress { 192.0.2.10/24 }
    track_script { monitaria_active }
}

Everyday use

• Make changes on the active collector. The standby copies its settings whenever they change, and its sensor states, acknowledgements and maintenance windows every third heartbeat. Anything changed on the standby is replaced at the next copy.
• Alerts about the pair go to the Catch-all webhook and to every rule in Settings › Alerts that isn't limited to device types or sites, labelled High availability: Down when the standby takes over, Warning when both were monitoring or the other collector stops answering, Up when it hands back.
• A takeover carries on from the copied states, so problems the primary already alerted about aren't alerted again, while recoveries are.
• On a standby that has taken over, Hand back now returns monitoring without waiting once the primary answers again. On a standby that is standing by, Copy settings now copies at once instead of waiting for the next heartbeat.
• Recent changes lists every takeover, hand-back and settings copy, and they're in the audit log too.
• The timings are on the same page: Check the other collector every (seconds), Take over after this many missed checks and Hand back after the primary has answered for (seconds). The defaults are 10 seconds, 3 and 120 seconds.
• Features the standby keeps switched off lists what stays off on the standby until it takes over: every receiver, poller, sender and schedule by default.

Good to know

• Messages between the two collectors are signed and encrypted with the shared secret, even over plain HTTP or a self-signed certificate, and go to no other host. Under More settings, tick Check the other collector's HTTPS certificate when both have certificates from your CA.
• Treat the shared secret like an admin password: whoever has it can copy this collector's settings, including saved credentials. It's write-only and never logged.
• Set a Witness, such as the default gateway 192.0.2.1 (or 192.0.2.1:443 to test a TCP port instead of ping), so a standby that is itself cut off doesn't take over. Without one, a network split leaves both collectors monitoring and alerting until it heals.
• History isn't copied. Measurements, events, NetFlow data and the audit log stay on the collector that recorded them, so the standby's charts start when it took over. The standby does refresh device details every 12 hours, so those are current.
• User accounts, API tokens, dashboards, the license and the HTTPS certificate belong to each collector. A pair is exactly two collectors: one primary, one standby.
• Settings that read a value from the environment (env:NAME) need that variable set on both collectors.