Ping not working from WAN interface

This seemed really strange; in a test configuration, where the “WAN” connection for OPNsense was in fact an existing (but not OPNsense) packet-filtering firewall, the WAN interface was happily being allocated an IP address by DHCP but it wasn’t possible to get a ‘ping’ response from the next-hop Gateway. Looking at the packets on the network, it was clear the ICMP packets were indeed being sent – and responded to – but these replies were never being seen by the ‘ping’ command. As a consequence of ‘ping’ not working, the status monitoring on the WAN interface was failing, considering it to be always ‘down’ – and so not routing any traffic to it.

The solution came via this Reddit post, which says:

Check firewall > settings > advanced. Change the “reply-to on wan rules” to the opposite.

This is referring to the “Disable reply-to on WAN rules” tick-box, for which the additional help text says:

With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, you must disable this behaviour if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.

The OPNsense manual page for this setting is here. It’s not clear how Bridging is related to this – there is a Bridge interface for one of the LAN VLANs; maybe the Tunable configuration settings required for that Bridge change the behaviour for the WAN interface too?

From some further reading, it seems the problem only manifests itself when the next-hop gateway is itself a packet-filtering firewall, so it might not be an issue in a more realistic deployment scenario.

There’s an in-depth discussion in this OPNsense Community posts: Reply-to on WAN by default is bogus

possible DNS-rebind attack detected:

Admittedly it’s a slightly unusual deployment scenario to have one DNS server running in the Outbuildings, referencing another DNS server running in the House – where both the House and the Outbuildings are using RFC1918 ‘private’ IP address ranges. One consequence of this deployment pattern is that the Outbuildings DNS server treats the receipt of an RFC1918 IP address from its upstream DNS server as a “possible DNS-rebind attack” – because dnsmasq is configured (by OPNsense) with stop-dns-rebind in its configuration file.

There’s certainly a requirement for the Outbuildings’ DNS server to return IP addresses for hostnames such as unifi (so UniFi network equipment knows where to find its Controller) which trigger the ‘rebind attack’ error if the response comes from the House’s DNS server.

One option is to add a duplicate, local entry for unifi to the Outbuildings’ DNS server, since that avoids triggering the error (because the entry is ‘local’) – but it means maintaining a duplicate entry on both DNS servers.

Another option is to use the rebind-domain-ok=/unifi/ dnsmasq configuration syntax to specify domain names which should be permitted RFC1918 addresses. Since this construct is not supported by OPNsense’s GUI for dnsmasq, such configuration lines need adding to a custom .conf file placed in directory /usr/local/etc/dnsmasq.conf.d/ While this works OK it’s a bit of a pain to have to maintain entries for all hosts in the House.

A third option is to not to specify stop-dns-rebind in the dnsmasq configuration file (and hence turn off rebind checking completely, for the Outbuildings’ DNS server) – but this line is automatically added to the main dnsmasq.conf file by OPNsense and there is no GUI option to remove it. It appears that the same result can be achieved by adding the following in a custom .conf file – which effectively says that ‘all hosts’ are rebind-domain-ok:

rebind-domain-ok=

A related point is that none of the DHCP host address reservations should have the ‘Local’ box ticked, since doing so declares the specified Domain will only be served by the local DNS server – and never forwarded upstream.

In the end, the all-round better solution turned out to be to centralise DHCP and DNS on the House firewall, using DHCP Relay settings on the local VLANs in the Outbuildings. A side-effect of this is that the DHCP leases returned by the House’s DHCP server automatically reference the House’s DNS server, so there’s only one DNS server that ever returns RFC1918 addresses.

TCP Session Timeout

Kept noticing packets tripping the “Default deny / state violation rule” when on inspection they belonged to legitimate traffic – so they were clearly failing on the ‘state’ element of the check.

There’s an Advanced Setting for the Firewall of “Firewall Optimization” which is “Normal” by default; changing this to “Conservative” seems to be helping to avoid this traffic being blocked unnecessarily.

Checklist for Adding an Extra VLAN

Adding a new VLAN isn’t difficult but there are quite a few steps required, some of which must be performed in a specific order.

The first step is to allocate the configuration settings for the new VLAN, notably:

• The numeric VLAN ID
• The four-character alphabetic name (referenced as AAAA below)
• The IPv4 subnet
• The IPv6 ‘prefix’ tracked from the WAN interface

If this is going to be a ‘bridged’ VLAN (i.e. stretched between the two buildings / routers):

1. Create the VLAN device on the LAGG side of the House firewall
   • The Device name should be vlan01.nnnn where nnnn is the numeric VLAN ID, padded to 4 digits (e.g. 0080)
   • The Parent must be lagg0 (LAGG)
   • The Description should be be L_nnnn
2. Create the VLAN device on the BACK side of the House firewall
   • The Device name should be vlan02.nnnn where nnnn is the numeric VLAN ID, padded to 4 digits (e.g. 0080)
   • The Parent must be igb4 (BACK)
   • The Description should be be B_nnnn
3. Assign both of these VLAN devices (to L_nnnn and B_nnnn respectively)
4. Add both of these VLAN device assignments to the Bridge_Parents Group (under ‘Firewall’)
5. Create the Bridge device, choosing the VLAN device assignments as the Member Interfaces and adding a Description of: Bridge for AAAA VLAN
6. Assign the Bridge device, changing the name to just AAAA and being sure to Enable and Lock the interface
   • Set the ‘IPv4 Configuration Type’ to Static IPv4 and enter the IPv4 details further down
     • The interface address will normally be 172.16.nn.1 and the CIDR suffix will normally be /22
7. Set the ‘IPv6 Configuration Type’ to Track Interface and enter the relevant Track Interface ‘prefix’ allocated earlier
8. Add the Bridge device to the Local_VLANs Group (under ‘Firewall’)
9. Add the new Subnet to DNSMASQ
   • Add it to the list of Interfaces to be served
   • Add a DHCP Range for it, normally 172.16.nn+2.1 through 172.16.nn+3.254
10. Create the VLAN device on the LAGG side of the Outbuildings firewall
11. Create the VLAN device on the BACK side of the Outbuildings firewall
12. Assign those and add them to the Bridge_Parents Group
13. Create the Bridge device on the Outbuildings firewall
14. Assign the Bridge device but do not Enable it (but do Lock it)

APC UPS Monitoring

While there is Uninterruptible Power Supply protection on both of the OPNsense installations, those UPSes are only expected to cover a couple of seconds of outage while the Tesla Backup Gateway switches the whole site over to battery power. It therefore doesn’t seem necessary to configure a controlled shutdown of OPNsense when the UPS is about to shutdown because its battery is almost empty.

However, it can be useful to have visibility of UPS battery health which one of the UPS monitoring utilities can provide. As a long-time user of APS UPS devices I was aware of the APC UPS Daemon (apcupsd) but read some reports that Network UPS Tools is generally preferred – mostly because it is still being actively developed (whereas apcupsd is not).

Both utilities are available as OPNsense plugins but NUT proved problematic with a USB-connected UPS. The UPS was being recognised by the Kernel with no issue:

# usbconfig
ugen1.1: <EHCI root HUB AMD> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
ugen0.1: <XHCI root HUB AMD> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen1.2: <Root Hub Advanced Micro Devices, Inc.> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
ugen0.2: <Uninterruptible Power Supply American Power Conversion> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (0mA)

However there were error messages suggesting NUT was not able to auto-detect the USB-connected UPS, perhaps due to permission issues:

# upsdrvctl start
Network UPS Tools - UPS driver controller 2.8.2
Network UPS Tools - Generic HID driver 0.53 (2.8.2)
USB communication driver (libusb 1.0) 0.47
libusb1: Could not open any HID devices: no USB buses found
No matching HID UPS found
upsnotify: failed to notify about state 4: no notification tech defined, will not spam more about it
Driver failed to start (exit status=1)

A bit of research turned up some matching error reports – notably NUT Doesn’t Find USB Buses – with some pointers to workarounds based on setting Tuneables to stop the Kernel attaching its own USB driver.

Trying apsupsd instead, that ‘just worked’ and immediately showed a status page with a whole bunch of details retrieved from the UPS. Looks like that will be the better option – especially since both UPS devices are from APC.

OPNsense – Miscellaneous Hints and Tips by Marsh Flatts Farm Self Build Diary is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.