Is VFIO not safe?
That is debatable, there is no clear answer there. The closest to a real answer to this could be found in this discussion over at the Qubes forum: https://groups.google.com/forum/#!topic/qubes-devel/MeLYpHyLRHQ
Have a nice day,
Not safe. Your GPU firmware can be replaced by malware in the VM.