Document recovery procedure after compromise

Website
1

Continuing the discussion from apt-get upgrading security issue CVE-2016-1252:

To my knowledge, Qubes has never recommended any kind of recovery / reinstall related to any Qubes QSBs. What would be the criteria for such a recommendation?

  • critical security bug + non-targeted, rampant attacks in the wild
  • critical security bug + confirmed targeted attacks + you being a high-value target
  • any paranoid user
  • ???

We might need 2 recovery procedures (though it would be impossible to know which to apply):

  • Full-compromise recovery procedure (including dom0)
  • Limited-compromise recovery procedure (non-hypervisor, domU only)

In event of compromise, would not recommend preserving any persistent filesystems. Better to manually migrate individual files after full reinstall / template reinstall.

1 Like
2

It depends what kind of compromise we are talking about. The silent and deadly type we are concerned with cannot ever be discovered nor removed by usual steps.

However if you have a good feeling that you’ve been affected - such as updating a vulnerable apt over an Exit Node then you are probably hosed. In this case the recovery solution is simple but a bit drastic - throw out the hardware and get a new set.

2 Likes
3

Right.

There all sound reasonable.

Right. And also the recovery involving new hardware as well that HulaHoop mentioned.

Yes

Yes, however that is a critically dangerous and difficult part. Some malware attaches itself to other files. [Would be great if documented somewhere.]

While migrating these files and/or opening lets say an office file, the malware might spread again. Therefore back to square one.

This is common in the wild in the Windows world with executable files, however I haven’t heard at time of writing this happening in the Linux world. There may be tailored attacks, but no off-the-shelf malware to my knowledge yet. (Off-the-shelf vs tailored attacks / malware is imo also an important concept to document.)

( Computer Security Education - Whonix )

It’s a question on how far users are willing to go. I doubt many will take extreme measures.

Let’s say theoretically it’s a vulnerability that we know was used in the wild and that escalated up to hardware compromise, then we’d need to explain this to everyone.

On the other hand unless confirmed targeted attacks / you being a high-value target / paranoid user, we probably should not recommend extreme measures to regular users.

We can only stick to honestly providing all information and then let the user make a decision. Without overwhelming most users of course. An short summary can be provided at the let’s say security guide. And the less realistic stuff (such as hardware replacement), the full information on advanced security guide.

What do you think?

1 Like
4

There is no way to ever discover such a thing?

Wouldn’t updating over an Exit Node apply to most whonix users?

Do you believe it likely that those who updated prior to the security issue becoming known are most likely compromised?

While no system is perfect, was hoping that running qubes os might mitigate the need to throw out the hardware if possibly compromised.
Any thoughts?

5

Detection of off-the-shelf malware is a very hard problem. Conceptually a lost cause.

In case of CVE-2016-1252 there are no reports of this being used in the wild. So no off-the-shelf exploiting CVE-2016-1252 exists.

See this chapter: Computer Security Education - Whonix

Detection of tailored malware is even harder.

Off-the-shelf malware if widespread enough has a chance of being detected by a technical person. It may also happen that tailored malware gets detected by a technical person, but the combination of tailored malware and a technical person looking and being lucky may not happen so often.

As a non-technical person you don’t have great options. Either you can:

  • Spend a fear years to learn the technical skills (knowledge about operating systems, network protocols, package analyzer, programming, disassembly), then try your luck.
  • Pay a lot money to a very technical person trying to find malware on your system. (Figure out the salary / payment for a security researcher / malware analyst and pay be ready to pay them for a few months.)
  • Be a high value target, have a probable explanation / evidence why you think you are being compromised and happen to find a very technical person to volunteer finding the malware. Most likely only people who have a strong intrinsic motivation. Like, if Snowden thought he was infected by a hardware virus, he might find someone looking into it to then publically expose it.

Yes.

No.

Running everything inside VMs is a very reasonable approach. However, it also only highers the bar. Making it more difficult / expensive to compromise the whole system. It’s not a perfect system.

No distribution of Linux [or Xen, or…] like Debian, Qubes, BSD or whatnot can solve the issue of not needing to throw out the hardware. These are issues with hardware so they can only be really fixed at the hardware level. Software levels at best can provide workarounds.

The problem is that there is no hardware that consists of entirely Libre Software. The firmware being Libre Software as well as also the hardware being Libre Software. Also analyzing the firmware of hardware or at least wiping a maybe-compromised version and overwriting it with a most-likely-clean firmware is much, much harder than it ought to be. (research fun: how to reflash the BIOS without starting the computer [because then the malware would be active and could tamper with that cleaning process]; how to do the same for the firmware of your disk controller; CPU; Intel AMT; what else needs reflashing - That stuff is so difficult, that it’s cheaper to buy new hardware.)

See also State considered harmful - A proposal for a stateless laptop

Stateless computers do not exist but that would be required to exist to not need to throw away the hardware.

Everyone convinced now, that we need stateless Libre Software hardware? :slight_smile:

1 Like
6

This writeup is excellent. Please add it to the malware chapter on the wiki.

1 Like
7

Yes it is!

Thank you Patrick for taking the time to explain in detail.

1 Like
8

Added here:

2 Likes
9

Document recovery procedure after compromise
https://phabricator.whonix.org/T580

10

Great. I polished it some more.

1 Like
11

Moved Malware and Firmware Trojans into their own templates.

Thereby a standalone page with its own opengraph data could be created:

1 Like
12

I talked to Marek (Qubes lead developer) - in try to summarize it from memory in my own: Even if he was paid to look several months into it. if not knowing what kind of backdoor one is looking for, it is just impossible to verify that a system is malware free. There is just too much to it.

Any outgoing protocol / communication just needs a tiny modification such as different delays (that are otherwise somewhat random) are enough to encode unwanted outgoing information leaks.

1 Like
13

Did we ever document recovery procedure after compromise?

14

Related:

16

Are you still up to document this? @HulaHoop

https://phabricator.whonix.org/T580

17

Done please review and close ticket if approved.

1 Like
18

It at hello? What’s the instructions about?

Various levels of compromise? Assume VM compromise, host compromise or hardware compromise?

Shut down compromised VMs/hosts.
Don’t boot compromised VMs/hosts.
Don’t power on compromised hardware ever again.

If you want to boot compromised something, make a backup of any storage medium first. This is because once booted, malware could delete/modify (more) data.

Connecting USB to host is only of if host is considered to be uncompromised. Otherwise malware could infect the USB hardware (firmware part, not storage) and spread elsewhere.

Rather than boot VMs, mount their images. Do the VM image mounting inside a VM to prevent the image mounting compromising the host.

Plaintext files are the easiest and safest to recover. These shouldn’t just be copied to another machine and opened there using a usual editor. Better to copy and paste its contents from one VM to another. This is to get rid of any malicious code attachments hidden in the plaintext file.

Even if mounting an image or harddrive and copying a file to another non-compromised computer or VM could result in the non-compromised computer or VM getting compromised. This is because once a VM or computer is compromised it can modify any file. And maliciously modified files can compromise programs that process them. For example a word document opened by office is not unlikely to compromise the operating system where its running. In theory even copying a compromised file could exploit a theoretical vulnerability in the copy command cp, not sure how likely that is.

Open files with a different editor then previously to lower the risk the adversary modified the file in a way it could compromise another VM or host if reused there.

Conversion of complex file formats such as word to plain text may significantly lower the chance of compromise when reusing the data.
Another example a jpeg file should be converted to bitmap which is a storage intensive but simpler data format which has a lower probability of compromising the viewer program. Then the user could optionally convert to jpeg to save disk space.

Anything from Compromise recovery in Qubes OS | Qubes OS that Qubes unspecific, good advice for users?

19

Whoops. OK this only concerns: Host compromise and/or Gateway comrpomise whcih can then root the host since it is part of the TCB and can access host services without firewall protection thru NAT networking. That is the situation to non-Qubes Whonix at least. What about Qubes?

Good point. Do you know how this is done? It will involve connecting the device though. So is not mounting it enough?

Yes. Was this not clear? It shouldn’t propagate if USB redirection is used properly.

I guess this is useful if you care about recovering from compromised VMs only, but the easier way for this is to simply transfer the files you care about out thru shared folder and delete it.

Interesting. Please add. What do you mean by code attachments? Is there an example we can point to?

True enough. This is why I said after transferring everything from the infected drive via shared folder the VM should be rolled back and the infected device no longer used. I think my writing must be terribly mangled that you missed that :frowning:

Sanitizing files is not foolproof and so I consider it out of scope. Even “simple” formats like RGB can carry crafted payload that might trigger something upon reconversion to another complex format. Easier to consider all files (except plaintext) as not trustworthy again.

The focus of this guide is to recover VMs. My approach is to consider all VMs and the compromised host hardware as expendable. It is also applicable to more setups/Whonix ports and even systems in general.

We can still mention it for those who want to do the VM recovery approach (Qubes users only)

1 Like
20

HulaHoop via Whonix Forum:

Whoops. OK this only concerns: Host compromise and/or Gateway comrpomise whcih can then root the host since it is part of the TCB and can access host services without firewall protection thru NAT networking. That is the situation to non-Qubes Whonix at least.

No, it can’t access any host services. VM breakout exploit required.

It’s a reasonable compromise to sometimes consider a Whonix-Workstation
compromised but nothing else.

What about Qubes?

Same from theoretic point of view but much stronger separation by
virtualizer. So no different documentation required regarding various
levels of compromise.

Good point. Do you know how this is done?

Malware is just a program that can do whatever it wants. There could be
a logic “if my runtime after start on this machine is low (so probably
shutdown after infection) do more damage by deleting files”. Or another
logic “if not internet access for too long, start deleting files”.

It will involve connecting the device though. So is not mounting it enough?

Mounting is for sure far safer than booting.

Yes. Was this not clear? It shouldn’t propagate if USB redirection is used properly.

I don’t think any users heard of USB redirection. And if that works,
once we consider USB firmware compromise then such a USB controller
would have forever not be connected to any VM or host that isn’t
supposed to be compromised.

The only way USB could be used to take data from a compromised VM is if
provided to a VM as a blockdevice without exposing the USB
controller/firmware like Qubes is doing. But in case of Qubes, no need
for USB, qvm-copy would do.

I guess this is useful if you care about recovering from compromised VMs only, but the easier way for this is to simply transfer the files you care about out thru shared folder and delete it.

Easier yes, but shared folders require the VM to be booted. And booting
a VM is far less secure than mounting.

If booting a VM: make a copy first in case the malware starts to destroy
files. Also disconnect internet from the VM first so the malware cannot
load new modules (with new exploit code) or do other malicious stuff online.

Interesting. Please add. What do you mean by code attachments? Is there an example we can point to?

There’s non-malicious pdf files and malicious pdf files. Any
non-malicious pdf file can be converted into a malicious pdf file by
modifying it. I am calling this sloppy “code attachment”. Perhaps better
“malicious modification”: Pdf readers are known to have sometimes
vulnerabilities reported about them. Maliciously crafted pdf files can
exploit pdf viewers, i.e. execute code even if pdf files shouldn’t
result in code execution.

There’s been also vulnerabilities found in text editors already, perhaps
even in connection with text files. I am not sure it happen in practice
yet or if it happen only on Windows yet. Search engines indicate it
happened in the Windows world. Nevertheless, theoretically it’s
possible. Parsing a pdf file and parsing a text file to show its
contents is basically the same, code parsing data, just that parsing a
pdf file is much more complex. This is because text editors interpret
the contents of text files. The more feature rich the text editor, the
more likely some parsing of the text file can lead to a code execution
vulnerability.

Another example would be cat. There’s been vulnerabilities in
terminal-emulators before. The simplest example of a malicious text file
would be if the victim will use cat on a text file and exploit a new
terminal-emulator vulnerability.

More sophisticated malware is known to infect (make malicious
modifications) to other files so a reinfection is likely once someone
reinstall and then reuses their data. That should be pointed out.

Sanitizing files is not foolproof

Indeed.

and so I consider it out of scope.

I see. Well, I doubt many users who go through the pain of compromise
recovery would go through the pain of trashing all of their data and
starting fresh.

The focus of this guide is to recover VMs. My approach is to consider all VMs and the compromised host hardware as expendable. It is also applicable to more setups/Whonix ports and even systems in general.

We can still mention it for those who want to do the VM recovery approach (Qubes users only)

I don’t think VM recovery is Qubes specific.

21

Confirmed. No services or ports from the host are visible via NAT network.

Considering the complexity and the number of things that can go wrong with USB, I changed the guide to use Onionshare which is much safer and manageable for users. I also tried to simplify the workflow of dealing with suspected files. Opinion?