Selfhosted

Lots of comments already mentioning the differences. I have tried these, including the mentioned ipfire, and decided on the end to use opnsense plus openwrt on two different devices.

I chose opnsense at the time many years ago because it supported wireguard out of the box, where as pfsense required some weird install process I didn't want to deal with. Plus I liked the UI to opnsense more.

My moden has been literally replaced by my firewall so I have the ONT connected to it and then use it to do all the heavy lifting for... Well, firewall stuff. It connects to a VPN so my entire network routes through the VPN. Then my openwrt device is connected to that. It also handles firewall stuff, but more at an internal level (keeping network devices only permitted to communicate with devices I say are okay, blocking internet access, etc) and also hosts my nginx setup to route to various servers.

While I could do everything on one machine with opnsense, I've got a particular setup that allows me to have multiple devices at the firewall level, truly isolated from the rest of my internal network (for a couple of internet open port services). And it gives me peace of mind that if someone found a zero day in opnsense, I'm not totally screwed unless they also got one in openwrt.

To answer "which is better to begin with", I personally find opnsense way more flexible and robust than the other 2 options. Has a lot more capabilities and upgrading is super easy without requiring jumping through weird hoops and such like openwrt does.

This thread has reminded me that I have Ruckus APs that mesh. But support had been dropped because they are "old". Presumably there is no open source solution that I can flsh these with, still allowing me the meshing?

pfSense and OPNsense are firewalls. OpenWRT is router firmware. They're all open source - to varying degrees - and they all have overlapping features and functionality.

Quick breakdown:

• OpenWRT: originally developed as a replacement for the firmware on Linksys wireless access points. It has grown into a full Linux-based networking OS with extensible features and broad hardware support. The target devices are still mostly wireless routers/access points and the use cases it services are still mainly about wireless networking.
• pfSense: Originally a fork of m0n0wall, it's a BSD-based firewall distribution. Designed primarily for firewall use cases, it can be loaded on bare metal or in VMs, but it's generally deployed "upstream" from wireless devices - typically it's the device that all of your network traffic passes through on the way in/out of the LAN. Extensible architecture and a rich ecosystem of plugins means that pfSense can also serve as a caching proxy, load balancer, intrusion detection server and logging host.
• OPNsense: a fork of pfSense. Almost identical use cases. OPNsense has a more usable/modern UI, but lags slightly in support for new features and plugins.

So the question of pfSense or OPNsense is either/or - you'd typically pick one or the other. Note that I'm staying away from the political comments that will invariably come up around this comparison. It's enough to know that both have commercial offerings in addition to their open source versions and people have strong opinions one way or the other.

Either one of either pfSense or OPNsense in conjunction with OpenWRT is common, with OpenWRT on the wireless devices and pfSense/OPNsense at the egress to WAN. In your case, Omada already does what OpenWRT would do - along with some very limited versions of what you could do with pfSense or OPNsense.

It's worth noting that folks often deploy these three open source tools as a method to regain control rather than using a third party cloud based solution like Omada. No judgement, just saying that Omada is the polar opposite of the 'selfhosted' esthetic.

Open vs closed solutions

I also like how OpenWRTs implementation of 802.11r doesn't require any central controller

Edit: the closed solution I'm referring to it TP-link