Without the extraneous 50-qxl.conf file that probably around zero VirtualBox are using, everything works fine. If it was a bug, it would be futile. "X11 does not work without this 50-qxl.conf file which we have for KVM compatibility but without it that works."
If there is a bug here, it's the X11 bug not doing proper auto detection and to require 50-qxl.conf:
Please explain how the build system works.
Most importantly, first everything is installed into the raw image before the hypervisor specific steps create specific vdi and qcow2 images.
Does it build two separate images in parallel (one for each hypervisor flag) from the same set of downloaded packages?
No parallel stuff.
If so then the --qcow2 flag would ideally omit any VBox packages and vice versa in the others image.
--target qcow2 only installs KVM stuff indeed and vice versa for --target virtualbox. But. Official images are created with both flags at once... About the why, I wrote that in post 4: https://forums.whonix.org/t/non-qubes-whonix-13-0-0-1-0-x-issues/2443/2
(Yes, some things could be improved. And speed up. In theory. Such as not building all Debian packages twice for gateway and workstation builds. And more. But it would require major work on the build script. And add more complexity. With the scare amount of people who completely understand and work on the build script as of now, I am hesitate to make it more complex. Perhaps some existing complexity could be buried by offloading it to other projects and/or Debian. For example contributing something like apt-get-unattended to Debian. Or something like chroot-raw could be generalized, turned into a command that gets merged into an existing package. Or one could contribute to Debian to get an official mechanism to build and distribute raw, vdi, qcow2 etc. images. And more. All of that requires manpower.)
This will become more necessary at some point when hypervisor specific packages like grsec kernel images are shipped.
I don't see hypervisor specific packages like grsec kernel images happening for many reasons but that's off-topic.