By "remotely accessible", do you mean remotely accessible to everyone or just you? If it's just you, then you don't need to setup a reverse proxy. You can use your router as a vpn gateway (assuming you have a static ip address) or you can use tailscale or zerotier.
If you want to make your services remotely accessible to everyone without using a vpn, then you'll need to expose them to the world somehow. How to do that depends on whether you have a static ip address, or behind a CGNAT. If you have a static ip, you can route port 80 and 443 to your load balancer (e.g. nginx proxy manager), which works best if you have your own domain name so you can map each service to their own subdomain in the load balancer. If you're behind a GCNAT, you're going to need an external server/vps to route traffics to its port 80 and 443 into your home network, essentially granting you a static ip address.
If this server is running Linux, you can use autossh to forward some ports in another server. In this example, they only use it to forward ssh port, but it can be used to forward any port you want: https://www.jeffgeerling.com/blog/2022/ssh-and-http-raspberry-pi-behind-cg-nat