Whonix vulerable due to missing processor microcode packages? spectre / meltdown / retpoline / L1 Terminal Fault (L1TF)

Known issue:

http://forums.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion/t/free-bios-hardware-that-support-all-your-needs/6411/9

Issue back in (at least) Qubes Debian based VMs or a false positive. Reported here just now:

spectre-meltdown-checker saying vulnerable inside Debian based Qubes VMs · Issue #4262 · QubesOS/qubes-issues · GitHub

1 Like

Some of these mitigations are available “optionally.” I mean like in the case of Vbox, as an example, the L1Df vuln can be mitigated on a per machine basis on commandline using “VBoxManage modifyvm…”
There are 4 total commands specific to this particular vuln, and documentation says “perf degradation can be SEVERE…”
I have these options enabled in my machines, including Whonix and do not notice any performance issues. Perhaps others would? I guess the chip makes a difference?
Commands: --l1d-flush-on-vm-entry on, --l1d-flush-on-sched on, --ibpb-on-vm-entry on, --ibpb-on-vm-exit on.
There is also a set of commands for other vulns that I have not enabled. According to logs on non-Whonix Debian virtual machines, with the above commands enabled, spectre v1 and v2 are mitigated, MDS is mitigated using a clearing of cpu buffers, but warns for full mitigation to turn off smt (hyperthreading). It lists Spec. Store bypass a still vulnerable though. Since Spec. Store Bypass has been mitigated on the host through a microcode update, is the VM still vulnerable? I am not sure.
On my host(s), everything is mitigated.
A copy of the vm logs:
Jan 28 20:33:32 host kernel: Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Jan 28 20:33:32 host kernel: Spectre V2 : Mitigation: Full generic retpoline
Jan 28 20:33:32 host kernel: Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Jan 28 20:33:32 host kernel: Speculative Store Bypass: Vulnerable
Jan 28 20:33:32 host kernel: MDS: Mitigation: Clear CPU buffers

EDIT: Here is a good list of mitigations that @Patrick posted some time ago:
https://github.com/Whonix/Whonix/blob/master/build-steps.d/2600_create-vbox-vm

Observations:
I notice that MDS mitigates along the same lines in the VM as on the hosts. Like if you mitigate mds on host, the virtual machine follows that and displays the same mitigation in the virtual machine. The Speculative Store Bypass does not for some reason, despite being mitigated on the host. Interesting though that for MDS, there is an associated VBoxManage command, 2 of them to clear on vm entry and/or on vm exit. There is also a --spec-ctl on | off option as well which I do not have enabled on the VMs.
On the Debian VMs, the only vulns listed as active is Speculative Store Bypass, and something called itlb_multithit according to /sys/devices/system/cpu/vulnerabilities. There is an entry for tsx_async_abort but it says “Not affected” inside the file. Obviously disabling hyperthreading on the host would take care of itlb_multihit. When it (hyperthreading) is disabled, the Debian VM is not vulnerable anymore.


As far as VirtualBox is concerned… Quote Spectre Meltdown - Whonix

There is no solution for VirtualBox yet. The bug has been reported to the VirtualBox developers. See VirtualBox 5.2.18 vulnerable to spectre/meltdown despite microcode being installed [archive]. The Whonix ™ developers depend on the VirtualBox developers for fixing this VirtualBox issue. Nevertheless: […]

Yes. Enabling those VirtualBox spectre/meltdown related security settings makes a VirtualBox Whonix-Gateway VM notifibly slow and the fan of my otherwise powerful notebook turns on.

@Patrick, I thought it might; thanks for the clarification

Interesting, creating a “40_spec_disable.cfg” in the /etc/default/grub.d folder with the spec_store_bypass_disable=on configuration and then updating grub showed on a reboot that it had no effect. The log still reported:
Speculative Store Bypass: Vulnerable
(The host is appropriately patched and not vulnerable)

1 Like

anontor via Whonix Forum:

Interesting, creating a “40_spec_disable.cfg” in the /etc/default/grub.d folder with the spec_store_bypass_disable=on configuration and then updating grub showed on a reboot that it had no effect.

Shows up in cat /proc/cmdline?

The log still reported:
Speculative Store Bypass: Vulnerable
(The host is appropriately patched and not vulnerable)

Virtualizer?

1 Like

Sorry about that! of course some details would help:
This is in Virtualbox 6.0.16. (on a Debian Buster host machine; also same result on the Ubuntu host)
cat /proc/cmdline shows:
BOOT_IMAGE=/boot/vmlinuz-4.19.0-6-amd64 root=UUID=26ceb4c2-0512-1428-347a-cbab4216a5c1 ro console=tty0 console=ttyS0,115200n8 intel_iommu=on amd_iommu=on slab_nomerge slub_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt spec_store_bypass_disable=on
Initially I tried the Gateway, and then Workstation, and both reported the same message

I can only point to my previous post here, specifically VirtualBox 5.2.18 vulnerable to spectre/meltdown despite microcode being installed. Feel free to reproduce this on Debian and take this up to VirtualBox as per Free Support Principle.

1 Like

I am certainly going to ask them. Speculative Store Bypass is the only one that states “Vulnerable” on boot according to logs. Everything else
reporting as mitigated including L1tf which has its own command which I enabled, and it also reports as mitigated.
This is what I have tried so far. Results were the same on both Whonix and a “regular” not-modified Debian 10 virtual machine. Results were the same whether microcode was installed on virtual machines or not. (Both hosts are microcode updated)
–spec-ctl on This option did what it is supposed to in that spectre variants were mitigated. No effect on Spec. Store Bypass. Also, if you do not enable this option and the host has been mitigated, the virtual machine will also be mitigated.
Grub commandline options: spec_store_bypass_disable on | auto | prctl | seccomp: I tried all of these options to no effect. One at a time with update-grub command and a reboot. Tried them with --spec-ctrl on and --spec-ctrl off and the results were the same: no effect on Spec Store Bypass. Still reports as vulnerable.
I will ask Virtualbox if they have any insight and see what they tell me. I would like to get this fixed. I cannot figure out why everything else is mitigated but not this? (On both a Debian 10 host and an Ubuntu 18.04 host, all vulns are reporting as mitigated through different methods.)

1 Like

Yes. Please report to VirtualBox. Better they hear it from others than just me bugging them.

I don’t know what to make off #17987 (VirtualBox 5.2.18 vulnerable to spectre/meltdown despite microcode being installed) – Oracle VM VirtualBox. First, VirtualBox developers implement spectre/meltdown defenses. Lots of effort. Then after a bug is reported, it’s suddenly not a priority anymore at all. Maybe #17987 (VirtualBox 5.2.18 vulnerable to spectre/meltdown despite microcode being installed) – Oracle VM VirtualBox is disregareded as such spectre-meltdown-checker oddity.

Please see #17987 (VirtualBox 5.2.18 vulnerable to spectre/meltdown despite microcode being installed) – Oracle VM VirtualBox and prepare what they’ll be asking for. Debian version, kernel version, VirtualBox log, kernel boot parameter (cat /proc/cmdline, cat /proc/version) for both host and VM, VirtualBox settings commands, vulnerability status host and guest.

for f in /sys/devices/system/cpu/vulnerabilities/* ; do echo "$f" ; cat "$f" ; done
1 Like

I will prepare that for them.
Your ticket gave a lot of info, and was re-opened. And then they just left you hanging. I have not heard or seen on their forums any more information about this. This is important, and I would like them to fix it.
I like that tiny bash script-- very handy!

1 Like

After extensive research and searching, along with some trial-and-error, I have come to a new conclusion about Vbox.
The version I tested is the most updated version in the 6.0- series which is: 6.0.16 from the vbox website. Two hosts were tried: Debian Buster and Ubuntu 18.04. Both hosts have had the proper Spectre/Meltdown mitigations through kernel updates as well as microcode. No vulnerabilities reported for either host. Checked /sys via a small bash script Patrick posted and also ran spectre-meltdown-checker and verified with log output after boot.
Now on to the virtual machines. Results were the same regardless of host. Results were the same for a Debian 10 non-Whonix virtual machine, and also the Whonix machines. 12 vulnerabilities are checked across the spectre/meltdown landscape b y the spectre-meltdown-checker tool v 42. Two CVEs reported as vulnerable: CVE-2018-3640:K0 and CVE-2018-3639:K0. These are referring to the Speculative Store Bypass issue and “Rogue system Register Read”. All other vulnerabilities checked for by the tool are successfully mitigated.
The options I have enabled for the virtual machines are as follows and can be enabled by the VBoxManage modifyvm “vm name” . --ibpb-on-vm-entry, --ibpb-on-vm-exit, --l1d-flush-on-sched, --lid-flush-on-vm-entry. These are set to “on.” There is a mds related command that I do not have set. The virtual machines report as mitigated. --spec-ctrl is set to “off.”
Regarding the Spec. Store Bypass, I found this: virtualization - VirtualBox & Spectre V4 - Information Security Stack Exchange. He believes the the SSBD flag has not been passed to the virtual machine(s) This is something that Virtualbox would have to enable. On the hosts, Spec Store Bypass is disabled explicitly by prctl and seccomp, which is supported in the Debian kernel. It should be showing up as mitigated.
Also, regarding the same issue, there is this current ticket at Virtualbox, #18477. In it, a user with CentOS and Fedora 29 reports that despite host mitigation, the virtual machines still report as vulnerable. This ticket is active right now. I was thinking of adding my experience on to this existing ticket as opposed to opening another, just because my issue is literally identical.
So, it appears that since version 5.2.18 up until now with version 6.0.16, progress has been made. If they can fix this last issue with the remaining CVEs, spectre and meltdown will hopefully be completely patched.

1 Like

You could also try VirtualBox 6.x series but I haven’t seen anything in the changelog regarding this subject.

now added:
Template:VirtualBox Spectre Meltdown: Difference between revisions - Whonix

1 Like

Awesome!
I am glad that the options exist for these various mitigations, as far as Virtualbox is concerned. As I was testing various configurations, and options it became clear that even without some options enabled, the guest still reported as mitigated (in the same manner as the host(s)). I suspect the microcode updates were the reason.
I had tried the most recent series of Vbox when it was 6.1. At the time, there were some problems with graphics-- specifically the sizing and full screen settings for VBoxSVGA and VMSVGA were buggy. I did not test for the presence of Spectre/Meltdown though. I notice that some graphics troubles seem to be fixed according to the changelog.
I would hope that the vulnerability defenses were at least carried over from 6.0.xx
I am going to keep a close watch on this Spec.Store Bypass issue, and any news about it will be shared with the Whonix forums. Virtualbox has its issues, as we know, but it has the potential to be excellent. I am going to do my best according to my skill level to help it become that for the Whonix community.

1 Like

Could you please also experiment with Whonix VirtualBox Paravirtualization - Which Acceleration Mode is Optimal? Help Wanted! - that might also help fixing this issue.

1 Like

Sure I can do that. i will review each choice for a full day and then come back to this thread and edit this post with the results.
I’ll start with the “KVM” virtualization method since it is the Virtualbox default for Linux hosts and the one I happen to be using.
Here are some findings following your linked post:

  1. Host OS Debian 10 (4.19 current kernel, updated fully)
  2. Host arch is 64 bit x86
  3. Whonix 15, updated fully (regular mainline, not the testers repo)
  4. Virtualbox 6.0.16 as obtained from The Vbox website (through their repository)
  5. Observations. Usability is smooth. No issues noted. There were graphics issues in the past that have been worked out. I do not use the 3d acceleration (there are some concerns about compromises that could take advantage of memory errors and allow unwanted access to host’s memory pages). I have tried it, but do not really have a noticeable difference.
  6. Youtube will work. Obviously make sure the Tor shield is partly open to allow some scripts. Also, click through and allow playback when NoScript warning appears. Make sure sound is set up on host and in Vbox and it will work fine. Some exit nodes are problematic but switch circuits to fix this.
  7. I have to test VLC; I have not yet, but I will.
  8. output: kvm-clock
  9. output: kvm-clock tsc acpi_pm
  10. I have iommu enabled for all virtual machines. Also, I help mitigate the time issues with: bootclockrandomization on all machines, tirdad on all machines, host system set to UTC also running bootclockrandomization, tcp timestamps turned off on all systems, no ntp or systemd-timesyncd on any systems including host.
    These commands are specific to Virtualbox:
    VBoxManage setextradata “VBoxInternal/VMMDev/0/Config/GetHostTimeDisabled” 1
    VBoxManage modifyvm --biossystemtimeoffset - (and +).
  11. dmidecode shows all Vbox related information, although the real chipset is identified in the virtualbox log files.
    cpuid command is unreliable, but can identify the real chipset for some values, for others it is not accurate.

EDIT: More Hypervisor “interfaces” to report

This is for the Virtualbox choice of “Minimal.” Vbox version remains the same, as well as the two tested hosts, and guests. Guests worked with were: Debian Buster vanilla guest machines, and both Whonix machines.
1-5 remain the same as above.
6. This is tough. There seem to be ridiculous amount of trouble to get on to the site. I changed circuit about a half-dozen times, that failed. Then I shut down the browser (Tor Browser) and restarted. This time it was successful.
7. VLC was fine. I actually used it to watch something from youtube, by putting the youtube video url into VLC and having it connect and play the video on VLC’s player. No problem noted.
8. output: tsc
9. output: tsc acpi_pm
Note: as we know, tsc is the clock that is on the hardware/cpu. is this better than kvm-clock? Probably it is, but not sure why exactly. Once again, the previous time-related mods were present for this test. In journalctl output, ioapic is reported along with its pointer in memory. Also, clocksource reported as: tsc-early using “pmtimer” reference calibration. x2Apic reported that the irq remapping does not support x2 apic (?). This is different than with “Default” or “KVM” which do not display that in journals.
11. Same result for dmidecode. Cpuid was similar, but the section on tsc and cpu frequency was very busy! It had a complete list of instructions that exist on the host’s actual chipset. Here, x2APIC was reported as active and had some stats about it. Seemed to contradict journal, but i have seen this from cpuid before with hypervisors. It can get confused.
Next will be “Legacy” for the Virtualbox hypervisor interface.

Legacy:
1-5 are the same once again
6. Youtube is great, no issues whatsoever, worked on first try.
7. VLC also works
8. output: tsc
9. output tsc acpi_pm
This was very similar to the “Minimal” option. The journal was almost exactly the same, timesource was reported as tsc once again. Dmidecode was boring. All Virtualbox information, nothing about the real chipset. Cpuid was similar to “Minimal,” but there were differences. For instance, tsc was noted, but that’s all. No memory information. Apic on chip was recognized and the journal had the same clock/timer readings:
x2apic: IRQ remapping doesn’t support X2APIC mode
host kernel: …TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
host kernel: tsc: Unable to calibrate against PIT
host kernel: tsc: using PMTIMER reference calibration
As with Minimal, Legacy worked very well, no problems to report. On the Default/KVM setting, I had tried to run with Nested Paging disabled and performance was degraded to a point of being un usable. With both Legacy and Minimal, the same result was noted. For the tested settings so far, the cpu temperature (as taken on the host from /sys/) is very reasonable and stays almost exact among the different options. I have not found one interface choice to be better than another, except for the fact that the potential security concerns of kvm-clock are not present with Minimal/Legacy.
Next one up is Hyper-V. As far as I am aware, this is the Microsoft native hypervisor. I am not running a Windows host, so do not know what to expect.

Hyper-V
1-5 the same once again. 6 works well with youtube, VLC is no issue.
8. output: tsc
9. output: tsc acpi_pm
10. I guess I was expecting some sort of windows-esque experience, but Hyper-V was almost the same as Legacy, and Minimal. I say “almost” because the journal produced this error that was not there on the previous two interfaces:
APIC calibration not consistent with PM-Timer and APIC delta adjusted to PM-Timer. One noteworthy item is that for every interface that used “tsc” as a clocksource, there was this same error in the early logs, right after boot: tsc: Unable to calibrate against PIT. It did not appear to cause any issues as in everything worked the way it should.
Dmidecode was the same, reporting all Virtualbox specific info: chassis, base board, bios, etc. Cpuid listed the hypervisor_id as “VBoxVBoxVBox.” Tsc was correctly identified, x2Apic was correctly identified, but some readings were just plain wrong. i guess these tools are really meant for testing on bare metal. The hypervisor seemed to really confuse the output of bot dmidecode and cpuid. As I was evaluating these different choices, I came across the Forum User : “Corrupt_Correct_Pig” It seems that a lot of what he/she reported was also true for me. So far, there is no difference in “looks,” as far as what the interface presents as. The menus look the same, and everything is identical. Nothing to give away what interface was chosen, other than checking out the choice in the GUI. I have not run into any issues that would prevent me from using one over the other. maybe a personal bias for a certain system, that’s it though.
There is one more left to check and that is the “None” choice. I will come back tomorrow to report on that one. I gave a day’s worth of activity on each choice. A day being 12hrs give or take.

1 Like

VirtualBox still vulnerable.

[…]
Just to summarize the current situation: The mitigation for the Spectre/Meltdown issues documented in CVE-2017-5715 can be passed through to VirtualBox guests using:

VBoxManage modifyvm --spec-ctrl on

This functionality is available in VirtualBox 5.2.32 and later, 6.0.0 and later, and 6.1.0 and later.

The VirtualBox changes required for passing through the Speculative Store Bypass (SSB) (CVE-2018-3639) mitigations and Rogue System Register Read (RSRE), Variant 3a (CVE-2018-3640) mitigations to VirtualBox guests have not been implemented yet.