In modern times it's important to keep in mind that the stiffness can be different independent of the type of filling.
That is, I'd pick filling primarily by allergies, and then by personal preference. They all feel marginally~moderately different, and some you'll just associate with "nicer". For me personally that's memory foam, for example.
Then independent of that, pick a stiffnes and, importantly, a shape that works for you. I sleep worlds better on a very stuff shaped pillow that perfectly matches my neck size and length.