I started facing the same issue as the OP. Never experienced it under Whonix 17 either. Following the general tips you linked didn’t fix it. I don’t know if it can be fixed, just wanted to inform that this is not an isolated issue.
I’m affected by the same issue and was able to capture the actual kernel oops via a serial console. It is a regression in the Debian kernel: 6.12.74+deb13+1-amd64 works fine, 6.12.111+deb13-amd64 crashes inside the vmwgfx driver.
So the reboot is caused by kernel.oops_limit=1 (expected Whonix hardening behaviour) — the real bug is the oops in vmwgfx command buffer allocation from the DMA pool during a screen-target plane update. The faulting address has its lower 32 bits zeroed, which looks like a use-after-free / memory corruption of the DMA pool free list (my guess).
At driver load the usual message is present:
vmwgfx 0000:00:02.0: [drm] *ERROR* vmwgfx seems to be running on an unsupported hypervisor.
vmwgfx 0000:00:02.0: [drm] Using command buffers with DMA pool.
Regression
6.12.74+deb13+1-amd64 — stable, no reboots with the same VM settings and the same Firefox usage.
6.12.111+deb13-amd64 — crashes after 2–10 minutes.
So something between Debian 6.12.74 and 6.12.111 broke vmwgfx under VirtualBox VMSVGA.
Workarounds tried
Disabling 3D acceleration: was already disabled — crash still happens.
VBoxSVGA and VBoxVGA: no crashes, but the desktop is extremely laggy and practically unusable.
Booting the older kernel 6.12.74 via GRUB “Advanced options”: works perfectly. Currently using this, with the package kept via apt-mark manual.
Possibly unrelated
The [W] taint comes from a warning early at boot (12 s, before vmwgfx is loaded), while udev-worker was loading ahci and tirdad:
WARNING: CPU: 1 PID: 319 at kernel/rcu/tree_plugin.h:826 rcu_sched_clock_irq+0x7cb/0xb60
Modules linked in: ahci(+) ... vboxguest(OE) ... tirdad(OEK+) ...
CPU: 1 UID: 0 PID: 319 Comm: (udev-worker) Tainted: G OE K 6.12.111+deb13-amd64
Not sure if it is related; the VM runs normally for ~5 minutes after it.
I can provide the full serial log or test intermediate kernel versions if that helps narrowing it down. Should this also be reported to the Debian kernel team / upstream vmwgfx maintainers?
Yes. Best if reported upstream. Seems your AI agent did a pretty good job at diagnosing this. Maybe your AI agent can also figure out where to best report this upstream.
Is this still reproducible with latest upstream Linux kernel? Maybe your AI agent can set up a separate, secure test environment (on a spare computer and/or VM or rented server depending on your requirements).
Your AI agent may even be capable of writing a patch that may or may not be useful for upstream to fix this issue.
AI based bug reports may or may not be permissible according to upstraem policy.
As a stop-gap, and assuming the analysis posted by @redvine123 is correct, it might be worth trying other emulated graphics options in VirtualBox, like VBoxSVGA.
Update / resolution: the crashes are gone after updating the host VirtualBox to 7.2.20 r175154 (matching the Guest Additions version) together with the guest kernel 6.12.113+deb13. VMSVGA, 3D disabled, Firefox — stable so far.
So my earlier “kernel regression” conclusion was premature: 6.12.74 worked with the old VirtualBox, but the actual fix seems to be the VirtualBox update. I changed both at once, so I can’t fully rule out the kernel