I would vote for running jellyfish off the NAS with powerful hardware Intel 8th Gen and up you should be fine. I personally don't transcode video on my server only audio.
If I'm not sure about a video format working, I'll manually re-encode the video with handbrake to h265 or av1. I also have used handbrake to create lower resolution copies of the same movies to reduce bandwidth needs when roaming on LTE.
Why do you need to transcode video?