The goated tutorial
I simply followed this tutorial, it does just work! Youtube: Jim’s Garage
A small clarification: The virtual IP should resign within the DHCP-range configured for the interfaces. The interfaces static IP should not be equal to the virtual IP or the IP of the peer. So in short:
| Devices | Interface | IP config |
|---|---|---|
| OPNSense (master) | LAN | 10.1.1.2 (static IPv4) |
| OPNSense (backup) | LAN | 10.1.1.3 (static IPv4) |
| virtual IP | LAN | 10.1.1.1 |
You can also find the official docs here: OPNSense - HA and Carp And a little diagram of mine:
Transclude of OPNSense-HA.canvas
Proxmox and multicast
Proxmox blocks multicast packets
This breaks the HA-”sensing” and will result in split-brain, if you are running these VM’s across different hosts!
You can have a look at this old article, but essentially: Proxmox and some switches/routers may block multicast to safe bandwidth.
Most likely IGMP performed by my switch or Proxmox itself is the source of these troubles. To learn more, Cloudflare has an article on it: What is IGMP snooping?
Solution
To circumvent this limitation, simply set the Peer-Addresses in the Virtual IP settings to the Peers IP-Address within the network (the static IP set in the interface settings).
Disable HA sync for Peer addresses
You have to disable HA-Sync for these settings, as the Peer-addresses would otherwise sync as well
IPv6 and HA
I don’t have HA setup for IPv6, since nothing about my IPv6 is static. Instead, the router advertisements of the master have a higher priority.
The help-texts of OPNSense/dnsmasq also advice to set the RA Interval and RA router lifetime to lower values to allow the clients to quickly notice if the router is down and switch to the one with the lower priority.
What exact values can be extracted from the help texts.
Without these settings, I had a fail-over-time of around 30 secs.
Disable HA sync for the DHCPv6 ranges.
Otherwise the router-priorities will sync, resulting in master and backup having the same priority
If you plan to expose a Webserver, ensure that your DNS entry has the correct IPv6 address. Your sever gets two GUA’s now, if both routers are active, this isnt a problem as they will just pass it all through. If one is offline, there might be a time where the DNS entry still shows to the old address, while the machine hasnt reported its new one back yet.
IPv4 and Webservers
Be sure to add the virtual IP as the NAT default route, otherwise all the HA isn’t worth anything :)
IPv4 and DHCP
You have to manually configure the DHCP Gateway since per default the Interface IP is send. This would break all routes, if the Interface goes offline :3
Some info on how to here:
- https://www.reddit.com/r/opnsense/comments/1ldha7c/migration_isc_dhcp_to_dnsmasq_how_to_use_custom/
- https://efficientip.com/glossary/dhcp-option/
TL;DR: 0. create a Tag (This step is slightly important)
- go into
Dnsmasq > DHCP Optionscreate a new entry for the Interface with these values:
| Label | Value |
|---|---|
| Interface | <your interface> |
| Type | Set |
| Option | router [3] |
| Option 6 | (idk, good luck!) |
| Tag | <Your Tag> |
| Value | <Carp IP> |
- assign the Tag to the DHCP range under
Dnsmasq > DHCP Ranges - PROFIT