homelab

My solution: Both

Opnsense should support HA. If you're using a vlan-capable switch, you can plug your ISP device into the switch and connext it to just these two machines.

By having a physical device, you get the stability advantages of a dedicated device. You can also test upgrades on the virtual router and roll back to the physical if needed. When something eventually goes wrong with the physical device (all hardware fails eventually), you fail over to the proxmox instance until you replace it and don't have to rebuild the config from scratch.

I currently have the exact same question in my head. I think I'll go the following route: Install opensense in a VM on my Proxmox host (it has 2 NICs) and just put my lab stuff behind it in it's own lan. Everything connects to the router via firewall.

Benefits:

• The rest of the lan (e.g. partner's devices) do not rely on my firewall working
• I don't need to buy anything, I can switch to bare metal later if I need to and have figured out what exactly I need

I ran pfSense on proxmox for a few years. It was fine, but unnecessarily complicated. I switched to an Intel n6005 mini PC and I'll never go back. Having a second device meant I was able to get rid of my Dell R720xd and switch to consumer hardware with no internet downtime. It means if something happens and I have to hard reboot my server, I don't have to worry about my partner getting booted from a video call. Etc. Etc. The mini PC was under $200. It sips power. It's silent. It's a no-brainer.

One big advantage with proxmox is that you can restore from your backup and have opnsense up again in few minutes.

I have 2 OPNsense, one on each Proxmox cluster Node, no problems (full 1Gbit/s throughput)

Go baremetal

You want it to be as simple as possible, to be as secure as possible.

Adding proxmox - or any abstraction layer - is now adding more layers that have potential security issues.

And everyone is scanning your IP for vulnerabilities 24/7.

Plus, in my case, I want a completely separate network for Guest Wifi, IoT, etc and only some stuff hitting the LAN / homelab.

A problem in proxmox means no router. Are you comfortable resolving issues without Internet access?

Pros: less physical hardware to deal with. If you can set up to where your VM can move across proxmox nudes, that improves resilience.

Cons: if you can't fail over, you could get to where you need to fuss with the box where the Opnsense VM lives and have to also take down Opnsense.

I've run OPNsense as a VM for a few years now. I have it set up on HA and have gone into PVE and noticed that it failed over and failed back without me noticing at all a week earlier. I like being able to snapshot it before updates, though updates are always flawless.

I have the 2 ethernet ports on each node named the same and that seems to work fine. I can also live migrate it without it dropping a ping in order to update the host node's OS, then migrate back.

I wouldn't do it any other way, but it might take some time to figure out how to set up so it fails over properly.

I followed this guide and have had zero issues. I had to do it this way because Opnsense didn’t natively support my 10g NIC. I have Proxmox handle the hardware side of things and pass through a virtualized card to Opnsense (albeit with slightly reduced performance).

In my home lab I have them separate the OPNSense box has full performance on its own HW, only needs to be patched once in a while and is super stable.

I have managed to crash / lockup one of my proxmox hosts at least once while messing around with HW past though or by giving a guest enough cores to slow the whole box down.

Family never gets interrupted playing games or streaming Netflix with my lab separate from the critical internet service.

New versions of OPNsense installed with ZFS support snapshots before upgrading natively sort of taking one of the promox vm tricks out of the pro list making it neutral.