Document recovery procedure after compromise

Yes it is!

Thank you Patrick for taking the time to explain in detail.

1 Like

Added here:

2 Likes

Document recovery procedure after compromise
https://phabricator.whonix.org/T580

Great. I polished it some more.

1 Like

Moved Malware and Firmware Trojans into their own templates.

Thereby a standalone page with its own opengraph data could be created:

1 Like

I talked to Marek (Qubes lead developer) - in try to summarize it from memory in my own: Even if he was paid to look several months into it. if not knowing what kind of backdoor one is looking for, it is just impossible to verify that a system is malware free. There is just too much to it.

Any outgoing protocol / communication just needs a tiny modification such as different delays (that are otherwise somewhat random) are enough to encode unwanted outgoing information leaks.

1 Like

Did we ever document recovery procedure after compromise?

Related:

Are you still up to document this? @HulaHoop

https://phabricator.whonix.org/T580

Done please review and close ticket if approved.

1 Like

It at hello? What’s the instructions about?

Various levels of compromise? Assume VM compromise, host compromise or hardware compromise?

Shut down compromised VMs/hosts.
Don’t boot compromised VMs/hosts.
Don’t power on compromised hardware ever again.

If you want to boot compromised something, make a backup of any storage medium first. This is because once booted, malware could delete/modify (more) data.

Connecting USB to host is only of if host is considered to be uncompromised. Otherwise malware could infect the USB hardware (firmware part, not storage) and spread elsewhere.

Rather than boot VMs, mount their images. Do the VM image mounting inside a VM to prevent the image mounting compromising the host.

Plaintext files are the easiest and safest to recover. These shouldn’t just be copied to another machine and opened there using a usual editor. Better to copy and paste its contents from one VM to another. This is to get rid of any malicious code attachments hidden in the plaintext file.

Even if mounting an image or harddrive and copying a file to another non-compromised computer or VM could result in the non-compromised computer or VM getting compromised. This is because once a VM or computer is compromised it can modify any file. And maliciously modified files can compromise programs that process them. For example a word document opened by office is not unlikely to compromise the operating system where its running. In theory even copying a compromised file could exploit a theoretical vulnerability in the copy command cp, not sure how likely that is.

Open files with a different editor then previously to lower the risk the adversary modified the file in a way it could compromise another VM or host if reused there.

Conversion of complex file formats such as word to plain text may significantly lower the chance of compromise when reusing the data.
Another example a jpeg file should be converted to bitmap which is a storage intensive but simpler data format which has a lower probability of compromising the viewer program. Then the user could optionally convert to jpeg to save disk space.

Anything from Compromise recovery in Qubes OS | Qubes OS that Qubes unspecific, good advice for users?

Whoops. OK this only concerns: Host compromise and/or Gateway comrpomise whcih can then root the host since it is part of the TCB and can access host services without firewall protection thru NAT networking. That is the situation to non-Qubes Whonix at least. What about Qubes?

Good point. Do you know how this is done? It will involve connecting the device though. So is not mounting it enough?

Yes. Was this not clear? It shouldn’t propagate if USB redirection is used properly.

I guess this is useful if you care about recovering from compromised VMs only, but the easier way for this is to simply transfer the files you care about out thru shared folder and delete it.

Interesting. Please add. What do you mean by code attachments? Is there an example we can point to?

True enough. This is why I said after transferring everything from the infected drive via shared folder the VM should be rolled back and the infected device no longer used. I think my writing must be terribly mangled that you missed that :frowning:

Sanitizing files is not foolproof and so I consider it out of scope. Even “simple” formats like RGB can carry crafted payload that might trigger something upon reconversion to another complex format. Easier to consider all files (except plaintext) as not trustworthy again.

The focus of this guide is to recover VMs. My approach is to consider all VMs and the compromised host hardware as expendable. It is also applicable to more setups/Whonix ports and even systems in general.

We can still mention it for those who want to do the VM recovery approach (Qubes users only)

1 Like

HulaHoop via Whonix Forum:

Whoops. OK this only concerns: Host compromise and/or Gateway comrpomise whcih can then root the host since it is part of the TCB and can access host services without firewall protection thru NAT networking. That is the situation to non-Qubes Whonix at least.

No, it can’t access any host services. VM breakout exploit required.

It’s a reasonable compromise to sometimes consider a Whonix-Workstation
compromised but nothing else.

What about Qubes?

Same from theoretic point of view but much stronger separation by
virtualizer. So no different documentation required regarding various
levels of compromise.

Good point. Do you know how this is done?

Malware is just a program that can do whatever it wants. There could be
a logic “if my runtime after start on this machine is low (so probably
shutdown after infection) do more damage by deleting files”. Or another
logic “if not internet access for too long, start deleting files”.

It will involve connecting the device though. So is not mounting it enough?

Mounting is for sure far safer than booting.

Yes. Was this not clear? It shouldn’t propagate if USB redirection is used properly.

I don’t think any users heard of USB redirection. And if that works,
once we consider USB firmware compromise then such a USB controller
would have forever not be connected to any VM or host that isn’t
supposed to be compromised.

The only way USB could be used to take data from a compromised VM is if
provided to a VM as a blockdevice without exposing the USB
controller/firmware like Qubes is doing. But in case of Qubes, no need
for USB, qvm-copy would do.

I guess this is useful if you care about recovering from compromised VMs only, but the easier way for this is to simply transfer the files you care about out thru shared folder and delete it.

Easier yes, but shared folders require the VM to be booted. And booting
a VM is far less secure than mounting.

If booting a VM: make a copy first in case the malware starts to destroy
files. Also disconnect internet from the VM first so the malware cannot
load new modules (with new exploit code) or do other malicious stuff online.

Interesting. Please add. What do you mean by code attachments? Is there an example we can point to?

There’s non-malicious pdf files and malicious pdf files. Any
non-malicious pdf file can be converted into a malicious pdf file by
modifying it. I am calling this sloppy “code attachment”. Perhaps better
“malicious modification”: Pdf readers are known to have sometimes
vulnerabilities reported about them. Maliciously crafted pdf files can
exploit pdf viewers, i.e. execute code even if pdf files shouldn’t
result in code execution.

There’s been also vulnerabilities found in text editors already, perhaps
even in connection with text files. I am not sure it happen in practice
yet or if it happen only on Windows yet. Search engines indicate it
happened in the Windows world. Nevertheless, theoretically it’s
possible. Parsing a pdf file and parsing a text file to show its
contents is basically the same, code parsing data, just that parsing a
pdf file is much more complex. This is because text editors interpret
the contents of text files. The more feature rich the text editor, the
more likely some parsing of the text file can lead to a code execution
vulnerability.

Another example would be cat. There’s been vulnerabilities in
terminal-emulators before. The simplest example of a malicious text file
would be if the victim will use cat on a text file and exploit a new
terminal-emulator vulnerability.

More sophisticated malware is known to infect (make malicious
modifications) to other files so a reinfection is likely once someone
reinstall and then reuses their data. That should be pointed out.

Sanitizing files is not foolproof

Indeed.

and so I consider it out of scope.

I see. Well, I doubt many users who go through the pain of compromise
recovery would go through the pain of trashing all of their data and
starting fresh.

The focus of this guide is to recover VMs. My approach is to consider all VMs and the compromised host hardware as expendable. It is also applicable to more setups/Whonix ports and even systems in general.

We can still mention it for those who want to do the VM recovery approach (Qubes users only)

I don’t think VM recovery is Qubes specific.

Confirmed. No services or ports from the host are visible via NAT network.


Considering the complexity and the number of things that can go wrong with USB, I changed the guide to use Onionshare which is much safer and manageable for users. I also tried to simplify the workflow of dealing with suspected files. Opinion?

in all cases: if starting something compromised: disconnect internet first

shared folder

Doesn’t apply to Qubes and could use links to existing documentation for virtualizer specific instructions.

Turn on your compromised machine

This shouldn’t be done or at least this should be considered a less secure option. With forensics and malware infections:

  • don’t continue to power on the hardware again
  • make an image and copy of that image before working with the image

Connect new external storage

I know what you mean but many users probably won’t. Needs to be clarified a bit what is considered new, i.e. not the ones connected to the infected host and why, to prevent BadUSB.

untrusted external USB or HDDs

Same as above. Untrusted probably meaning: no data stored on it that you care about or wiped beforehand, and knowing that this device can never be connected again to any non-compromised computer. Perhaps trusted/untrusted doesn’t describe it well? Perhaps better compromised/non-compromised or infected/uninfected? The previously uninfected HDD should be considered infected after being connected to a malicious machine?

Threat model: You know or highly suspect that the Workstation is compromised, but there was no VM breakout.

  1. Transfer your data out of the Workstation via shred folder.

If so: remove internet connection first.

But then still applies: It is easier but not the most secure option.

Threat model: You know or highly suspect that the Gateway is compromised, but there was no VM breakout.

A compromised Gateway has no access to the host resources. Only the Workstation should be considered compromised along with any data that it contains or is in shared folders it can access.

Why should the workstation be considered compromised? By mistake wrote workstation but meant gateway?

Why not? Considering exfiltration has already happened what are we protecting the infected machine from? If not the network then the HDD Needs to be taken out and put in a USB enclosure to be connected offline from the clean machine. This brings up another set of problems that can also lead to infection. At least with the network we can have higher chances of keeping the new machine trustworthy. Also the forensic method is not accessible to most people.

OK

Yes I understand these are reasons for not powering on compromised devices but if something was to auto destruct then the damage could have been done before you’ve even received back the devices or even accessed them again. If that was the intention of the attacker.

OK I will make clearer

In this context, the data was exposed but the machine not infected because of whonix-firewall. Data exposed because the GW can see all traffic.

1 Like

Risk of booting a malicious VM without internet connected:

  • if internet is considered off for too long: malware might react on this trigger such as by deleting files or maliciously editing files (adding fabricated evidence)
  • or do some action to damage the hardware

Risks of keeping internet connected when being compromised:

  • malware can load more code and the newer code might include stuff for infecting more hardware or VM breakout
  • exfiltration may not be a binary yes/no. Could be something in between. Imagine a malware where the attacker remotely is searching one by one (there are screenshots/videos available about malware features). Not all attackers straight fetch a whole image dump. Malware might upload more files: upload over Tor is slow. Depending when one noticed the compromise, specifically with lots of data (lets say video recordings) not everything may be in the hands of the attacker yet.
  • the attacker might upload illegal files (non-torified) somewhere that result in a police raid
  • the attacker might come up with a new strategy such as uploading false evidence files to the victim’s computer and then tip the authorities resulting in a police raid and false accusation
  • when a malicious computer is connected, it might do nefarious work such as for a spam botnet
  • the longer it’s connected to the internet (image a user following that advice and taking days for searching all the data and making backups) it’s harder to explain (think in front of a judge) that “my computer was compromised, it wasn’t me” vs “your computer was used in an attack for days”.

See this image to see what compromised computers are used for: https://krebsonsecurity.com/wp-content/uploads/2012/10/HackedPC2012.png

You’ll want none of this. So: don’t connect a compromised VM or machine to the internet.

You’re right. Sounds damn awful.

I will include your post on why it’s not a good idea to do this.

As a workaround that does not involve USB or internet, one can install syncthing on both endpoints and transfer via LAN only. What do you think?

1 Like

Edited again. Please check.

1 Like

It’s still in opposition to “don’t power on compromised machines”.

Also looks like an attractive attack surface to compromise the new machine.