Document recovery procedure after compromise

[entr0py]

Continuing the discussion from apt-get upgrading security issue CVE-2016-1252:

apt-get upgrading security issue CVE-2016-1252

I don’t think we have documented a full compromise recovery procedure anywhere.

To my knowledge, Qubes has never recommended any kind of recovery / reinstall related to any Qubes QSBs. What would be the criteria for such a recommendation?

• critical security bug + non-targeted, rampant attacks in the wild
• critical security bug + confirmed targeted attacks + you being a high-value target
• any paranoid user
• ???

We might need 2 recovery procedures (though it would be impossible to know which to apply):

• Full-compromise recovery procedure (including dom0)
• Limited-compromise recovery procedure (non-hypervisor, domU only)

In event of compromise, would not recommend preserving any persistent filesystems. Better to manually migrate individual files after full reinstall / template reinstall.

[HulaHoop]

It depends what kind of compromise we are talking about. The silent and deadly type we are concerned with cannot ever be discovered nor removed by usual steps.

However if you have a good feeling that you’ve been affected - such as updating a vulnerable apt over an Exit Node then you are probably hosed. In this case the recovery solution is simple but a bit drastic - throw out the hardware and get a new set.

[Patrick]

To my knowledge, Qubes has never recommended any kind of recovery / reinstall related to any Qubes QSBs.

Right.

What would be the criteria for such a recommendation?

• critical security bug + non-targeted, rampant attacks in the wild
• critical security bug + confirmed targeted attacks + you being a high-value target
• any paranoid user

There all sound reasonable.

We might need 2 recovery procedures (though it would be impossible to know which to apply):

Right. And also the recovery involving new hardware as well that HulaHoop mentioned.

In event of compromise, would not recommend preserving any persistent filesystems.

Yes

Better to manually migrate individual files after full reinstall / template reinstall.

Yes, however that is a critically dangerous and difficult part. Some malware attaches itself to other files. [Would be great if documented somewhere.]

While migrating these files and/or opening lets say an office file, the malware might spread again. Therefore back to square one.

This is common in the wild in the Windows world with executable files, however I haven’t heard at time of writing this happening in the Linux world. There may be tailored attacks, but no off-the-shelf malware to my knowledge yet. (Off-the-shelf vs tailored attacks / malware is imo also an important concept to document.)

( https://www.whonix.org/wiki/Computer_Security_Education#Malware )

---

It’s a question on how far users are willing to go. I doubt many will take extreme measures.

Let’s say theoretically it’s a vulnerability that we know was used in the wild and that escalated up to hardware compromise, then we’d need to explain this to everyone.

On the other hand unless confirmed targeted attacks / you being a high-value target / paranoid user, we probably should not recommend extreme measures to regular users.

We can only stick to honestly providing all information and then let the user make a decision. Without overwhelming most users of course. An short summary can be provided at the let’s say security guide. And the less realistic stuff (such as hardware replacement), the full information on advanced security guide.

What do you think?

[eujan1]

It depends what kind of compromise we are talking about. The silent and deadly type we are concerned with cannot ever be discovered nor removed by usual steps.

There is no way to ever discover such a thing?

However if you have a good feeling that you’ve been affected - such as updating a vulnerable apt over an Exit Node then you are probably hosed. In this case the recovery solution is simple but a bit drastic - throw out the hardware and get a new set.

Wouldn’t updating over an Exit Node apply to most whonix users?

Do you believe it likely that those who updated prior to the security issue becoming known are most likely compromised?

While no system is perfect, was hoping that running qubes os might mitigate the need to throw out the hardware if possibly compromised.
Any thoughts?

[Patrick]

There is no way to ever discover such a thing?

Detection of off-the-shelf malware is a very hard problem. Conceptually a lost cause.

• http://arstechnica.com/security/2014/05/antivurus-pioneer-symantec-declares-av-dead-and-doomed-to-failure/
• A botnet authors was even writing undefeatable malware and trolling antivirus vendors. - https://www.reddit.com/r/IAmA/comments/sq7cy/iama_a_malware_coder_and_botnet_operator_ama/?limit=500000

In case of CVE-2016-1252 there are no reports of this being used in the wild. So no off-the-shelf exploiting CVE-2016-1252 exists.

See this chapter: https://www.whonix.org/wiki/Computer_Security_Education#Malware

Detection of tailored malware is even harder.

Off-the-shelf malware if widespread enough has a chance of being detected by a technical person. It may also happen that tailored malware gets detected by a technical person, but the combination of tailored malware and a technical person looking and being lucky may not happen so often.

As a non-technical person you don’t have great options. Either you can:

• Spend a fear years to learn the technical skills (knowledge about operating systems, network protocols, package analyzer, programming, disassembly), then try your luck.
• Pay a lot money to a very technical person trying to find malware on your system. (Figure out the salary / payment for a security researcher / malware analyst and pay be ready to pay them for a few months.)
• Be a high value target, have a probable explanation / evidence why you think you are being compromised and happen to find a very technical person to volunteer finding the malware. Most likely only people who have a strong intrinsic motivation. Like, if Snowden thought he was infected by a hardware virus, he might find someone looking into it to then publically expose it.

Wouldn’t updating over an Exit Node apply to most whonix users?

Yes.

Do you believe it likely that those who updated prior to the security issue becoming known are most likely compromised?

No.

While no system is perfect, was hoping that running qubes os might mitigate the need to throw out the hardware if possibly compromised.

Running everything inside VMs is a very reasonable approach. However, it also only highers the bar. Making it more difficult / expensive to compromise the whole system. It’s not a perfect system.

No distribution of Linux [or Xen, or…] like Debian, Qubes, BSD or whatnot can solve the issue of not needing to throw out the hardware. These are issues with hardware so they can only be really fixed at the hardware level. Software levels at best can provide workarounds.

The problem is that there is no hardware that consists of entirely Libre Software. The firmware being Libre Software as well as also the hardware being Libre Software. Also analyzing the firmware of hardware or at least wiping a maybe-compromised version and overwriting it with a most-likely-clean firmware is much, much harder than it ought to be. (research fun: how to reflash the BIOS without starting the computer [because then the malware would be active and could tamper with that cleaning process]; how to do the same for the firmware of your disk controller; CPU; Intel AMT; what else needs reflashing - That stuff is so difficult, that it’s cheaper to buy new hardware.)

See also State considered harmful - A proposal for a stateless laptop

• https://blog.invisiblethings.org/2015/12/23/state_harmful.html
• https://github.com/rootkovska/state_harmful/blob/master/state_harmful.md

Stateless computers do not exist but that would be required to exist to not need to throw away the hardware.

Everyone convinced now, that we need stateless Libre Software hardware?

[HulaHoop]

This writeup is excellent. Please add it to the malware chapter on the wiki.

[eujan1]

This writeup is excellent.

Yes it is!

Thank you Patrick for taking the time to explain in detail.

[Patrick]

Added here:

• https://www.whonix.org/wiki/Computer_Security_Education#Malware
• https://www.whonix.org/wiki/Computer_Security_Education#Hardware_Torjans

[Patrick]

Document recovery procedure after compromise
https://phabricator.whonix.org/T580

[HulaHoop]

Great. I polished it some more.

[Patrick]

Moved Malware and Firmware Trojans into their own templates.

• https://www.whonix.org/wiki/Template:Malware
• https://www.whonix.org/wiki/Template:Firmware_Trojans

Thereby a standalone page with its own opengraph data could be created:

Malware

[Patrick]

I talked to Marek (Qubes lead developer) - in try to summarize it from memory in my own: Even if he was paid to look several months into it. if not knowing what kind of backdoor one is looking for, it is just impossible to verify that a system is malware free. There is just too much to it.

Any outgoing protocol / communication just needs a tiny modification such as different delays (that are otherwise somewhat random) are enough to encode unwanted outgoing information leaks.

[Patrick]

Did we ever document recovery procedure after compromise?

[Patrick]

Related:

Compromise recovery on Qubes OS