Already some time since I tried Whonix. The feature I still use on a daily basis is the Live Mode (of the host) via Grub-Live. Now I got the idea that I want to verify for myself that the live mode is doing what I always hoped it was doing: preventing persistance of changes to my disk. So I want to make a system image, do a checksum, work on the system, re-image and recalculate the checksum and test for equality. Ideally I should not have to ask you how to go about this, on the other hand maybe I do not fully understand what I am doing, so I ask you: is this a good approach to testing if the live mode is indeed doing what I think it is doing?
Can I make such an image with clonezilla on a usb stick? What utility do you recommend for the checksums?
See this chapter:
Anti-forensic Claims
- Create an image of the disk using
dd. [1] - Checksum it with any tool. sha256, sha512, …
- Boot into host live mode.
- Create another image of the disk using
dd. - Create another checksum.
- Compare the checksums.
That way the whole disk can be compared. Including changes to files / things which might be overlooked (bootloader, partition table, free space, deleted files not in any manifest).
If checksum was different, we’d only know that something changed. But not exactly what. In that case, indeed it might be easier to have checksums (or even copies) of all individual files.
[1] Could be deleted after creating checksum if you’re on low disk space but then later if there where differences, it couldn’t be compared what changed.
Ok, I put Debian on an usb stick, booted from that usb, checked with lsblk that the drive I wanted to check (sda) was unmounted, and dd’ed an image to a second usb-drive. Then I exited, booted again like normal in my ‘live mode user’ in my main drive (sda), did some work (1. sudo apt upgrade, 2. deleted some /va/log files 3. connected to a vpn, and then to the internet 4. emptied the trash in Files 5. wrote a LibreOffice text, 6. connected an encrypted external usb drive and that was about it). I shutdown. Booted again with my usb-stick, checked that sda was unmounted, dd’ed a second image to the external usb drive. Took a sha256sum from both images, compared, and found… that they were not equal.
I wrote up my doings quite literally so you can see that probably I did not make a procedural mistake. The checksums should have been the same, but weren’t. (I must say that sda when unmounted is encrypted, but that is not the problem, is it?)
So, what can I do? I cannot really live with this state of affairs. But compare all individual files seems a too daunting task. There must be a more clever way to go about it. Divide the files in user files, system files, other divisions maybe. I ask what you think might be the most fruitful way of dealing with this. Because this may potentially be a very time consuming investigation.
Separate /boot partition?
/boot in separate partition not covered. Tested just now. Created a file /boot/test which persisted after boot. Therefore also not covered because these are symlinks to /boot.
/initrd.img/initrd.img.old/vmlinuz/vmlinuz.old
/home/user/test and /etc/test was gone after reboot.
Probably also not covered:
- Partition management tools such as gparted.
- Changes to partition table.
- Changes to bootloader (
grub-install).
sudo update-grub and sudo update-initramfs -u do not work in live mode. These shouldn’t result in modifications on the disk.
Encryption might make analysis harder but goal is to make that work of course too.
Good, because if the disk gets (auto) mounted and modified, then of course checksums would be different. Since encrypted however, I don’t think there could be any auto mounting. Well, except for /boot but I haven’t heard of any auto mount + auto modifications.
Reproduce the issue without involving the grub-live package for simplicity and to get access to upstream support.
grub-live is just a means to easily setup existing functionality created by upstream.
At time of writing it installs the following packages as per debian/control file:
live-boot live-boot-initramfs-tools live-tools
Most importantly, installs this file:
And eventually, file debian/grub-live.postinst effectively results in running:
sudo update-grub
That’s it. For anything more, try contacting upstream.
It has to be scripted of course. That reduces the manual labor. Replaces impossible manual labor with medium difficulty scripting. It’s been done. [1]
Maybe “just” the “apt-update” broke it as said above. Because if you create/modify a file in /etc/ or /home/user/ folders in live mode, these are gone after reboot, right?
vbindiff might give insights.
[1] analyze_image was previously used to reduce non-determinism (related to reproducible builds / verifiable builds) in two different Whonix builds. It creates all the sha512sum files and more in a report folder. When running this twice (for different images), these can be compared (such as using meld and/or kdiff3).
analyze_image requires an unencrypted image. cryptsetup luksOpen would probably suffice.
Hi Patrick, thanks for responding. I am reading your answer several times, but have some difficulty understanding what you mean.
-
You say: Separate /boot partition? You suggest that persistent changes occur in /boot, so by creating a separate /boot partition, we can more easily establish that the rest of the system remains unchanged?
1b) my whole system is now in one big partition. Can I rearrange that setup with my current system or does a separate /boot partition require that I reinstall my whole system anew? -
Reproduce the issue without involving the grub-live package. Haha, yes, nice. You mean I should somehow get a ramdisk loaded like what you have in de grub-live menu entry: linux /vmlinuz-4.19.0-16-amd64 root=/dev/disk/by-uuid/d89995f1-7a5f-4292-b0d5-8ec000b06184 ro boot=live plainroot union=overlay ip=frommedia noeject nopersistence quiet
Just so that upstream will not report the case back to you… But I am a bit in the dark how I should accomplish that.
-
The file debian/grub-live.postinst. Where can I find that file?
-
Maybe “just” the “apt-update” broke it. Ok, that is a good first starting point. I can check it again but this time no apt updates. I will do that tomorrow.
-
If persistent changes in “live” only occur with apt update/upgrade I might conclude that the changes do not mean a security hazard, right?
-
Lastly I might try analyze_image. I still have the 2 images by dd. Can I ‘cryptsetup luksOpen’ these two, you mean, running from my separate usb-debian?
- You say: Separate /boot partition? You suggest that persistent changes occur in /boot,
Yes.
so by creating a separate /boot partition, we can more easily establish that the rest of the system remains unchanged?
No.
I am just saying /boot isn’t protected from modifications.
1b) my whole system is now in one big partition. Can I rearrange that setup with my current system or does a separate /boot partition require that I reinstall my whole system anew?
I don’t think you don’t have a separate /boot partition. In other words:
you probably already have a separate /boot partition. Recommended to
check with partition manager gparted. Actually non-trivial to encrypt
/boot. You need to follow a manual to do that. Not a default feature
unfortunately yet anywhere as far as I know.
- Reproduce the issue without involving the grub-live package. Haha, yes, nice. You mean I should somehow get a ramdisk loaded like what you have in de grub-live menu entry: linux /vmlinuz-4.19.0-16-amd64 root=/dev/disk/by-uuid/d89995f1-7a5f-4292-b0d5-8ec000b06184 ro boot=live plainroot union=overlay ip=frommedia noeject nopersistence quiet
By doing what the grub-live package does, you don’t have to manually
enter any uuid.
Just so that upstream will not report the case back to you…
But I am a bit in the dark how I should accomplish that.
In other words, while grub-live was a great contribution to simplify the
setup, it didn’t invent the Debian live packages mentioned in my
previous post. That’s upstream. That’s providing everything you need.
grub-live doesn’t give you anything you couldn’t get from Debian on your
own.
- The file debian/grub-live.postinst. Where can I find that file?
In the grub-live git repository.
- Maybe “just” the “apt-update” broke it. Ok, that is a good first starting point. I can check it again but this time no apt updates. I will do that tomorrow.
Great!
- If persistent changes in “live” only occur with apt update/upgrade I might conclude that the changes do not mean a security hazard, right?
Yes.
Related: root compromise could re-mount the disk read/write anyhow, do
selective changes. A related concept:
As per above concept, it would be more secure anyhow to only boot with a
user rights, unable to gain root.
Once sudo/root is used, grub-live can be circumvented through /boot and
other other things mentioned in my previous post (partition table).
Maybe someone (or upstream) has an idea how to protect /boot and/or
other things but ultimately without physical disk write protection it
will always be imperfect against sudo/root (use or compromise). That’s
already documented.
- Lastly I might try analyze_image. I still have the 2 images by dd. Can I ‘cryptsetup luksOpen’ these two, you mean, running from my separate usb-debian?
Once you have unencrypted dd images (or any (raw) image), analyze_image
can be used from anywhere. Even normal production system. There is no
need for a special environment. But there are some problems coming to
mind…
crytsetup luksOpen gives you “only” a /dev/mapper/… device. I am not
sure that’s essentially the same as a unencrypted raw image. Would guess
so. dd reading from /dev/mapper to file should give a real raw image but
might not be needed (saving disk space).
analyze_image was developed to analyze Whonix images. Whonix base images
are created using grml-debootstrap. Which doesn’t use multiple
partitions. I.e. creates no separate /boot partition. analyze_image
doesn’t support multiple partitions which are default when installing
using Debian (probably based too) installer(s). Therefore analyze_image
probably won’t work. analyze_image at time of writing lacks two features:
- encrypted images support (otherwise cryptsetup luksOpen is already
difficult) - multiple partition / separate /boot partition support
I am not aware of any other tool that can do that.
Your options:
- search if there is another tool for this task (reproducible builds
project might have such as sub project) - contribute to analyze_image
- wait for reproducible builds project to create such a tool
That latter could take years if not decades. I guess reproducible builds
project creates reproducible repository packages, then reproducible
installed packages, then reproducible Debian installer ISO and only
eventually reproducible Debian VM images. Why Debian VM images matter?
Because when comparing two host dd images (which are raw images) that’s
actually similar to comparing two different VM raw image builds.
An update. Re-image’d my system drive, and compared with checksum the same drive after having done some changes there. No apt update/upgrade this time. Unfortunately the checksums still don’t match.
Patrick, you are right that I do have a separate /boot partition.
What I can do now, what is easiest to do now, is to re-image the drive again, just now solely partition sda5 (from a linux usb drive, so sda5 still unmounted and encrypted). Is that a good idea?
What is sda2 being used for?
Regarding reproducing the issue without grub-live. I understand what you say about all the functionality already being there. But I don’t understand how and when I should give the kernel instructions to load itself ro and non-persistently. You don’t want me to write my own grub scripts, do you? So, how should I go about it?
Regarding analyze-image, it is not a real problem that the tool has no multiple-partition support, is it? We already established (well, you did) that /boot isn’t inmutable, so I can just dd 1 partition: sda5. And use that one in the analysis.
gparted might have a better idea. You might want to look that up elsewhere such as fdisk - Understanding partition table with sda1 sda2 sda5 - Unix & Linux Stack Exchange
I guess if you’d boot from USB then “sda becomes sdb” (maybe) (I guess you know that already). Then cryptsetup luksOpen and only doing a dd backup for the / partition seems a good idea indeed.
All steps are already in this post: Testing (to please myself) Live Mode User - #4 by Patrick
It’s just 3 steps to do the same that grub-live is doing. Packaging wise, it’s one of the simplest packages. A lot other files are basically “distractions” (unfortunately required files for licensing, packaging, and whatnot such as file COPYING). “Bloating” the number of files. Making things look more difficult than these actually are.
- install the packages:
live-boot live-boot-initramfs-tools live-tools
-
install the /etc/grub.d/11_linux_live
-
regenerate /boot/grub/grub.cfg
sudo update-grub
That’s it.
No need. It’s already there. Mentioned in linked post and this post.
Right, in theory that should work.
Ok, I got now two images (before and after activity) of the main partition sdx5. As an aside: I extracted them with dd from the earlier disk images by starting at the relevant sector. I thought that was clever but the amount of sectors in /boot is relatively small and the extraction from the disk image was about three times slower than the dd’ing of the unmounted file system before. But that is not important. What is important is that the checksums proved equal. 
So in effect it is just the /boot partition that caused the persistent changes in “live mode user” (ok, I did not test this second sdx2 partition). So, what should I do? I am not intimately familiar with what is stored in /boot. What matters to me most is that the delta changes in /boot do not reveal interesting forensics about my digital presence. I think I am justified in this hope. Do you agree? What I could do lastly, is do the analyze-image on the /boot partition. See what that comes up with.
Great!
Ideally /boot would be encrypted by default (when enabling full disk encryption) but Debian (any Linux distribution) isn’t there yet. Most people don’t encrypt /boot yet. See also discussions online whether encrypting /boot is worthwhile or not. Over simplified “most people don’t care about /boot”.
Knowing all of the above however, changes in /boot contains anything super sensitive after use of host live mode. At most it reveals that disk was used if any timestamps or something changed there. I’ve never heard that a browser history or some document backup got accidentally leaked there.
For “normal” use cases such as using host live mode with user user and then browsing, document editing, other things that don’t require root, there is no way for these non-root applications to write into /boot anyhow.
To ease my soul I investigated some more into the differences in de before and after images of my /boot partition (after having used the “live mode user” of grub). I tried to use analyze_image but that excited with errors without being very clear about the problem. So, really it wasn’t that hard to script a procedure to calculate of every file in the /boot partition a checksum (sha256sum) and compare the resulting reportfiles with diff (and/or cmp).
I extracted the /boot partition of my disk image, that being a file, loaded as a /dev/loop0 device, and mounted that one on /mnt (I tried also to use ‘meld’ here, but couldn’t get rid of the “file filters” so it was totally unclear to me what it did compare and what not, besides the presence of the “lost+found” directory caused an error). My little script of maybe 5 or 6 lines calculated the checksums of all the files in /mnt (really /boot). Diff did not find any difference in checksums of the before- and after- mounts.
So what were the differences about really? Not in de main partition sdx5, and neither in sdx1 (when mounted). It was diff (or cmp) that pointed to differences of the boot image (the image of the /boot partition). The first difference that came up was at byte 1069 or 0x42D. According to a data map of the ext4 filesystem (Ext4 Disk Layout - Ext4) is that byte one of 4 bytes starting at 0x42C that store the Mount time, in seconds since the epoch. (In the data map it is the 4 bytes starting at 0x2C, but we have to subtract 0x400 = 1024 bytes of the first reserved (unused) block. The superblock is the second block. Ok, the /boot partition is not protected from changes (not by the “live” mode anyway) and this is the first piece of data that changes. And with good reason, I understand why Mount time changes, and I find no security hazard here.
To cut some corners I compared again the before-and-after images of /boot, but this time I skipped the first four sectors (2 blocks, 2048 bytes) of that partition. Now diff and cmp do not find any difference anymore. So far I am pleased (with myself…). And thanks Patrick for pointing me to the right direction.
mount time (time of last mounting of partition) is a pretty good guess.
Maybe mount option in /etc/fstab noatime for /boot (and generally?) would help?
Maybe there is a kernel boot parameter similar to noatime that we could set?
Ideally in live mode mount times shouldn’t be recorded. It’s not a critical issue but for simplicity of dd whole encrypted disk and compare it would be much more handy if this issue wouldn’t exist.
Non-Qubes-Whonix (and Kicksecure) VM images do not have a separate /boot partition. Could could test the same there. I don’t see why grub-live in VMs should be for the purposes of tracking disk persistence issues would be much different than a real host. (Except for separate vs non-separate /boot partition.) Therefore it might be (more) comfortable to test using VMs.
If all files in /boot have the same checksums, what might have changed are file creation dates. (And much less likely: linux user permissions.)
Update to differences noticed in /boot in before-and-after images: the command cmp stops at the first difference it finds. So, with hexedit I gave 0x2C Mount time the same value, then cmp found a difference in 0x30 Write time and when I erased that there was a last difference in 0x178 Number of KiB written to this filesystem over its lifetime. Three differences I found that are not shocking but do reveal that someone was at a specific date/time behind the keyboard doing things.
I did not check the noatime option. It seems to result in no updates when files are merely read but not when written to. So I doubt it will help a lot.
About other solutions: the proliferation of user-roles (persistent user, live user, persistent untrusted root etc) I only use two flavours: the persistent user, who can sudo to root and apply system changes and the live mode of user. Mostly I only use live mode, in that mode you can sudo to root, but changes are not persistent. So I guess, a pretty safe configuration.
The separate /boot partition which is not protected from changes in live mode, and is not encrypted. For protection in live mode I should contact upstream. But encryption of /boot, is that a good idea, and how does it help? When you boot the system, /boot is no longer encrypted, right?
There is a feature, which gets into the way here:
It’s a puzzle. For live mode, we don’t want that feature. Can it be disabled?
If you can find a way to disable it, I’ll think about how to do this by default (in live mode). It looks like it’s not possible but please search more.
It’s related to the superblock.
- Mount time
It’s a puzzle because there are many ways in theory to solve it. Either by disabling recording of mount time. The only suggestion I found so far is “mount as read-only”. Another piece of the puzzle.
What options exist to mount /boot as read only?
cat /etc/fstab | grep boot
# /boot was on /dev/sda1 during installation
UUID=redacted /boot ext2 defaults 0 2
You could try to configure mounting /boot as read only in /etc/fstab. You might want to experiment dong that in a VM unless you would be able to undo this using Recovery Mode if that renders the system unbootable.
In another quest for more secure mount options in (re-)mount home [and other?] with noexec (and nosuid [among other useful mount options]) for better security? we haven’t implemented a good way yet as a Linux distribution on how to change mount options. Would help if /etc/fstab.d · Issue #12506 · systemd/systemd · GitHub was implemented.
So until (re-)mount home [and other?] with noexec (and nosuid [among other useful mount options]) for better security? is resolved which would change boot options permanently, I don’t think I’ll figure out how to change boot options dynamically, i.e. enable read-only for /boot when booting into live mode unless anyone contributes.
More puzzle pieces: How does the boot process work generally? Who/what mounts /boot? Initramfs? Where is the source code for that? I cannot easily find that out.
Maybe if Whonix changed to dracut (replacing initramfs-tools with dracut) mount related features ((re-)mount home [and other?] with noexec (and nosuid [among other useful mount options]) for better security? and mount /boot as read-only in live mode) would be easier to implement.
dracut’s man page mentions /boot. Maybe, hopefully dracut’s interface (config, modules) are flexible enough to implement this.
Roadmap:
- replacing initramfs-tools with dracut - #6 by HulaHoop
- (re-)mount home [and other?] with noexec (and nosuid [among other useful mount options]) for better security?
- mount /boot as read-only in live mode
It’s a matter of empathy. Successful communication. Trying to estimate what the reader will think.
“Hi upstream, can you please do same as grub-live but better?”
“Hi upstream, can you please do same as [something you’ve never ever heard before] but better?”
“Hi upstream, can you please do same as [something you’ve never ever heard before] but better?”
“Hi upstream, can you please do same as [a package with 16 files of which 3 are important] but better?”
“Hi upstream, can you please do same as [1700 lines of code in total package] but better?”
…that might end up with “lost at hello”.
The hope is, that upstream that provides “99%” of the functionality would be able to implement a feature request that would implement the “1%” functionality that grub-live is implementing, but better, with /boot mounted as read-only. Short of implementing it yourself, a well written feature request is your best chance.
For live mode / same hash of disk: doesn’t help.
Encrypted /boot generally: I don’t think I could provide a better answer than what is avaialble on search engines for search term “encrypted /boot”.
@Patrick, thanks as always for your answer
Yeah, I will see if I can do something with /boot mounting “ro”. Maybe you are right that I should try this in VM first 
The last couple of days I delved in memory forensics (with volatility) to see what they get when they capture your system while running…
When I know more I will post that here.
Trying to work my way to daylight. Booting with /boot “ro” worked. But I wondered if that always works, like when the filesystem-journal is dirty and needs to update, etc. I think I saw Whonix itself not having a separate /boot partition. And grub-live only protects one partition. So, maybe I should go back from 2 partitions (/boot and /) to just one big partition. So that grub-live overlays the whole filesystem including /boot. But of course my root partition / was already encrypted with LUKS2, so the question to be addressed was also can GRUB boot from an encrypted partition. Then I found this super-user-friendly Debian guide:
https://cryptsetup-team.pages.debian.net/cryptsetup/encrypted-boot.html
that instructed me exactly how to get an encrypted boot. And my hope was that by doing so, by concentrating everything in just one partition that grub-live would protect everything.
I replicated my partitionscheme in VM with Debian10 as OS. Followed the cryptsetup guide, got rid of the separate /boot. And yes that works, and yes when I choose grub-menu-entry Live mode user, changes to my /boot directory (directory now, not a partition) are gone by the next boot. So that is all very well: in VM at least a fully encrypted and fully by Live mode user protected system. The only reason why I will not implement this on my “real” system (non-VM) is the lack of GRUB support for LUKS2. To be able to boot from an encrypted partition I had to downgrade LUKS2 to LUKS1. So, there are advantages as well as disadvantages. So I reckon it is not worth the trouble.
It seems that this GRUB support for LUKS2 is lacking already for two years now.