Honestly, I'm absolutely happy with my Steam Deck, I think it ticks most of your boxes (it even runs Linux, so it's essentially a portable Linux computer designed for gaming), so I think it's the better option that you're looking gor. To your points specifically:

it's really geared towards family/party gaming

There are plenty of party games on Steam.

it's Nintendo, so you get the whole usual games (Mario Kart, Zelda, etc.)

This is the only reason to get a switch, if you want a Nintendo console and Nintendo games this is the way. Everyone who gets a switch understand this is the reason they're getting it. If this is as strong a point to you that it makes you overlook everything else, then get the switch.

like most consoles, it's plug and play and can be enjoyed in the living room (I kind of gave up trying to set up a proper gaming experience with my Linux PCs, given that I don't have the hardware for it)

Steam Deck also has a Dock that you can plug to your TV, you'll need controllers but even so it should be much cheaper in the long run since games are extremely affordable compared to Nintendo.

the battery life is not great to say the least (2.5 hours takes me back of the Game Gear in early 90s!)

Haven't seen many benchmarks of the switch to be honest, but that does sound bad, the Deck only gets that bad battery life if you're playing Cyberpunk or something, for more casual games it can get upwards of 6h. Plus you can get power banks that fast large it while playing, which I assume is also possible on the switch although the switch 1 used to have some issues with power banks.

the screen seems to be pretty bad too (at least it's a step back from the OLED one of the Switch)

All but the cheapest Deck models now use a 90Hz OLED panel

the joycons are still not using a Hall effect sensor, meaning they might still be prone to drifting

While the Deck's default sticks are not hall effect, they are easily replaceable and Valve sells hall effect replacements on ifixit, so if you ever get drift in your sticks it's fixable.

most of the games will not be sold as proper cartridges but as download codes

If you're going down this rote Steam sells download codes for much cheaper

the whole thing (console, additional gamepads, games) is quite pricey

The Deck is about the same price, but like I said you'll end up saving in games since you start with your whole Steam Library and can get more games much cheaper.

it's Nintendo, famous for their anti-everything (anti-homebrew, anti-emulation, anti-piracy)

The Deck is by far the most open console you can get, you can even replace the entire OS if you want to, but StramOS is great and you shouldn't need to.