They are generally large, uncompressable and replaced instead of updated like text files. All files stay in the repo history forever, they make repos big and slow compared to text files with no advantages provided (e.g. as you said, diffing etc is useless).

If a binary file needs to be stored in git, it's usually more appropriate to use git LFS for that file. Git LFS stores the binary outside of the repo in the same way that database engines store binary outside of the respective table.

In this case, it would be much smarter to use version control on the text in the document, not tte binary file, which is a feature of essentially every document writer program.