Qubes Is For You (a guide)

Qubes-Whonix
1

Qubes Is For You

There is a lot of noise on social media trying to convince you that Qubes is not for you. They say that before you use Qubes you have to read every manual, be expert in Linux, and accept a diminished experience.

Qubes 101 - read this before u even begin save yourself time!
https://old.reddit.com/r/Qubes/comments/11jamz6/qubes_101_read_this_before_u_even_begin_save/

Before you begin: If u are a gamer, qubes os is NOT FOR YOU If u have any passwords for any system of website that are 8 charactors or words with ou DoB on the end, QUBES IS NOT FOR YOU. … if you have got to this point here are my recommendations: (3 to 6 months of learning required) Dont install qubes, [instead] navigate to ubuntu OS website download it install it use it for a good month or 2…

Qubes OS Review
https://old.reddit.com/r/linux/comments/tjr0qx/qubes_os_review/

Given that this OS has a high technical barrier to entry, it isn’t suited for anyone unfamiliar with Linux or virtualization.

None of that is true.

Qubes is for you.

Qubes is for anyone who wants to take their computing environment seriously.

Qubes is for you if

  • you worry about digital banking risks and fraud
  • you’ve been targted by malware in an email
  • you handle sensitive or proprietary information
  • you have sensitive contacts
  • you want a better and more secure Tor experience
  • you’ve lost data due to viruses
  • you’re concerned about spyware
  • you’ve ever covered your computer’s camera

Qubes, designed around the idea of isolation, more or less solves the above issues. (Knock on wood.)

If you drop the soap: https://forum.qubes-os.org/t/memes-about-qubes-os/4422/151

Qubes isn’t going to replace your gaming or video production PC, because it isn’t great with GPUs.[^1] But it can replace your daily driver for ordinary computing tasks: email, messaging, coding, web surfing, writing, watching movies – anything CPU-based. And if you need better security, privacy, and anonymity, Qubes+Whonix is hard to beat.

But Qubes isn’t just secure. It’s also the best desktop virtual machine manager.

Here’s a word about that:

What Qubes OS actually is, is probably the best UI for VMs to date, along with so many security features implemented by default, that the user doesn’t have to feel like they’re “walking on eggshells” when doing anything on their machine. And, most importantly, if any aspect of those features are not to your liking, you can change them, which is absolutely glorious. https://forum.qubes-os.org/t/mental-outlaw-reviews-qubesos/15627/16

Qubes is also just a solid desktop experience.

[1]: If you’re determined, look up ‘GPU passthrough’ for more information about using GPUs with Qubes.

What You Need To Know

You don’t need advanced technical knowledge to use Qubes

You need to know how to install an operating system.[^2] Admittedly this is a hurdle, but Qubes is easier to istall than some secure operating systems (OpenBSD). (To make it even easier, I suggest you skip the PGP verficiation of the Qubes media the first time you install Qubes.)[^3] The biggest immediate issue is whether Qubes will accept the hardware. Sometimes it does, sometimes it doesn’t. In my experience, Qubes always installs. But it doesn’t always recognize crucial hardware (like networking).

These caveats aside, Qubes is easy to use. It is graphical: the installer is 100% graphical, the login is 100% graphical, all settings are available in graphical menus. It comes with sensible defaults, and works out of the box.

While it’s better to be familiar with Linux, you don’t need advanced knowledge of the Linux terminal to use Qubes.[^4] Is it different from Windows or MacOS? Sure. It is different from Linux as well. QubesOS is its own thing, and you have to learn how to use it. While you can use Qubes right away after the install, configuring Qubes so it works well for you takes an investment of time and effort. Every Qubes install is unique, because everyone has their own workflow. Over time you will learn, and if you stick with it you will not be disappointed.

[2]: Here are the basic steps: https://www.qubes-os.org/doc/installation-guide/
[3]: I get it, you think this is bad advice. Here’s worse advice: Jump through 100 hoops before you ever get to try Qubes, so you never even bother. You can always verify it the second time.
[4]: Linux commands you should know: pwd (present working directory) cd (change directory) ls (list files) nano (text editor); and how to get help: <command> --help or man <command> or help <command> or whatis <command>. In nano press to save, to exit, and to exit without saving.

Hardware for Qubes

You don’t need specialized and rare hardware to use Qubes.

The only required security features – virtualization (HVM) and IOMMU – are very common. The rest are optional.

Other than that, there is only one firm requirement: minimum 16GB RAM. Anything less is probably a waste of time. And if possible, upgrade your traditional harddrive to at least a SATA SSD. That’s a strong recommendation.

Instead of trying to squeeze your fancy PC and it’s GPU into a Qubes straight-jacket, just pick up a second machine for Qubes. You can build a Qubes machine for $500, because you don’t need a GPU.

Here are some core specs:

  • CPU: Ryzen 7
  • RAM: 32GB
  • SSD: 1TB NVME

Select a motherboard from the recommended hardware list, and fill out the above specs and you will have a reasonably fast Qubes system.

Or pick up a used business mini desktop for $200 or $300. That’s what I did. My first Qubes machine had limited specs (i5/16GB/HDD). By chance it had all the security features required by Qubes, including TPM.

Here is how to make a limited system work for you:

  • Manually reduce the memory usage of all qubes (See: Appendix: Installation Notes)
  • Design workflows that only require 3-5 qubes available at a time (it’s enough)
  • Accept that things are slow
    • This is not a laptop-at-the coffee-shop experience
    • Booting takes 15 minutes – grab a coffee
    • Opening and closing qubes take a minute – think ahead
  • That said, the application experience – web, writing, coding, watching videos – is perfectly acceptable.

But if you have a good machine? It’s fast. I even run local AI on my Qubes machine.

Security Features of Qubes

You don’t need all the features of Qubes to get security benefits.

Perfection is the enemy of the good. Don’t listen to people who say you shouldn’t do X or Y because it reduces your security. Just using Qubes puts you ahead of everybody else. As you learn more, you’ll make smarter decisions. And remember: not every user of Qubes has to be a super secret spy or journalist handling top secret documents. If you want to login to Google and Facebook on your Qubes machine, go for it!

Here are some great security features you get from Qubes even without hardware like TPM:

  • Qubes with imperfect hardware is better at compartmentalizing VMs than hosted solutions like VMWare and VirtualBox
  • Amazing VM workflow without the hassle of managing a bloated and vulnerable host environment
  • Secure cut and paste, with warnings if you’re pasting what looks like code into a terminal
  • Warns you when a link or file you clicked wants to open an application
  • Secure conversion of risky pdfs inside a disposible VM
  • Open risky files or media in a disposible VM
  • Whonix provides secure, Tor-always connections, and a hardened OS with lots of interesting security features
  • Automatically checks for updates across multiple operating systems
  • Color coded windows so you always know where you are
  • Community of people who can teach you

Adjusting to Qubes

You can learn Qubes

You will have to make some adjustments and think differently when using Qubes:

  • dom0 is the ‘king’ of the virtual machines
    • To access the dom0 terminal, click Q/<settings;wheel>/Other/Xfce Terminal
    • You can’t install anything in dom0
      • That keeps dom0 nice and sparkling clean
      • But you can backup from dom0
  • You open applications by opening qubes (virtual machines)
    • A different qube for every major activity
  • Learn how cut and paste work between qubes
    • Setup (in Qubes Global Config) to use <Win-c/v> keys, because <Ctrl-Alt-c/v> should be available to the terminal for cut and paste
    • Remember to use the clipboard menu (top right corner) to copy out of dom0
  • Setup keybindings that enable quicker access to core activities and qubes
    • See: Q/<settings;wheel>/System Settings/Keyboard Application Shortcuts
    • See: Appendix: Keybindings (below)
  • Learn how to setup qubes so they are efficient to use, and also reproducible in subsequent installs
    • Analyze your workflows and figure out what you want to compartmentalize
    • You will create 30 qubes the first time, reduce that to 18, and finally settle on 8 or 10
  • Learn how to organize your files inside each qube so that you can efficiently gather and backup information across multiple qubes
    • If you leave data laying around in random places, data recovery, backups, and reinstalling is going to be a hassle
    • See: Appendix: Backup Script (below)
  • Learn how to install software
    • For standard software / app qubes:
      • open Setup for that qube, click on Applications, add/remove applications
    • For other or custom software:
      • Apps don’t have persistent system storage, so apps installed via package managers like apt disappear between boots
      • Standalone VMs have their own persistent system storage, and thus you can install software on them
      • Install via package managers like apt, or build a development environment and compile and install using pipx <program> go install cargo install <program> or make install, etc.
  • Learn to use qvm-move-to-vm and qvm-copy-to-vm to move files between qubes
  • Learn how to attach and detach USB devices to and from specific qubes (an extra step)
  • Choose to limit which qubes you attach USB storage to, to limit the impact of malware
  • Learn where the setup and config options are
    • Start here: Q/<settings;wheel>/Qubes Tools > Qubes Global Config
    • See also: Q/<settings;wheel>/Qubes Tools > Qube Manager
  • Pick a color scheme for your qubes that works for you and maintain consistency
    • See: Appendix: Qubes Naming and Color Conventions
  • Wrap your head around global and local screenshotting
    • Global: in Keyboard, set xfce4-screenshooter -r to and save in ~/Pictures in dom0
    • Local: type <Ctrl-Shift-S> in either Firefox or Tor Browser to save local screen shots in Downloads
  • Full screen windows with <Alt-F11>
  • Information, setup, and config may not be where you expect
    • Logs are in dom0: /var/log/xen/
    • Diskpace is in the disk widget at the top righthand of the screen
    • etc.

QubesOS is Stable and Recoverable

Don’t worry about losing data.

Given the risks of viruses, malware, and spyware on other operating systems, your data is more secure on a Qubes system than probably anywhere else.

I’ve had one (early) fatal system crash, but I recovered all my data. My second install (same hardware) has lasted more than a year untill this day without a problem, and rarely crashes (maybe twice), and I reboot maybe once every couple months.

If your Qubes system fails, you can mount the drive on another Linux machine, so long as the LUKS encryption holds (it will). It’s not hard to search through the folders to find your data, especially if you put some thought into how to organize your qubes internally.

You can save your LUKS header with this command:

Command:
  $ sudo cryptsetup luksHeaderBackup /dev/<yourdevice> --header-backup-file <output-filename>
Example: 
  $ sudo cryptsetup luksHeaderBackup /dev/nvme0n1p3 --header-backup-file qubes-os-header.luks

To learn your storage device name, type lsblk in dom0 and look for the parent of luks-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
Mine is nvme0n1p3.

To help you, I’ve written a backup script so you can move important data our of your qube onto external media.

  • See: Appendix: Backup Script (below)

Any Problem?

You can solve it.

Inevitably, you will experience a problem that requires searching qubes <problem> in your favorite search engine and a trip to the terminal. You will learn; I did.

Thankfully there is sufficient community to get answers to most questions, and you can join for free.

The Qubes forum accepts a temporary email from a 10 minute email provider (search 10 minute email); The Whonix forum doesn’t require an email to sign up.

See the following helpful guides:

Appendix: Links to Qubes Documentation

Appendix: Installation Notes

  • Install options: Stage 1
    • Encrypt drive
    • Set locale (time zone) (I forgot and it messed up Whonix)
  • Install options: Stage 2
    • Add USB mouse support by default
      • Otherwise system will be unusable if debian update leads to sys-usb reboot failure (happens sometimes)
    • Install updates via Whonix
  • Managing Memory Usage
    • qubes default to 4GB each, which is a problem if you have 16GB RAM or less
      • Thankfully you can reduce the memory requirements of most qubes
      • On my 16GB system, I can run 5 qubes in addition to dom0 and the 4 sys qubes
      • These are not scientific numbers, but they work for me
    • Reduce Dom0 Memory
      • Open dom0 terminal
      • Edit grub
        • $ sudo nano /etc/default/grub
      • Change
        • From: dom0_mem=max:4096M
        • To: dom0_mem=max:1536M
      • Update grub
        • $ grub2-mkconfig -o /boot/grub2/grub.cfg
    • Reduce Sys-* Qubes Memory
      • In Settings/Advanced/Max memory
      • sys-firewall : 600M
      • sys-net : 500M
      • sys-usb : 500M
      • sys-whonix : 600M
    • Reduce Qubes Memory
      • In Settings/Advanced/Max memory
      • The biggest issue for the browser is memory being filled by large videos, complex pages with lots of extraneous media and scripting
      • If your qube is hanging, increase its memory and restart the qube
      • Tor Browser qubes : 1500 (few tabs) : 1800 (more tabs) : 2500 (many tabs)
      • Video / YouTube qubes : 1800 (few tabs) : 2500 (more tabs) : 3000 (many tabs)
      • Vault qubes : 400
      • Media player qube : 3000; untested but ‘makes sense’ if playing large HEVC/h265 files
  • Managing VCPUs
    • VCPUs are virtual CPUs, which can be in any number (virtualized by Xen), but objectively track the number of cores your CPU has (not threads)
    • You can set the VCPUs in the qubes setup menu under Settings/Advanced
    • Set all sys qubes to 1
    • Reduce all low computation qubes to 1
    • Playing video? Keep at 2, maybe 3 if you’re watching 4K videos.
    • Compiling software? Set to 4 or 6 or CPU maximum, and let Xen sort it out.
    • Running local AI? Set to CPU maximum.
    • All of the above advice depends upon the power of your CPU.
    • Generally speaking, don’t assign more VCPUs to a qube than you have cores, but YMMV.
  • Ergonomics
    • Turn off lock screen if you don’t need it: Click battery, turn on Presentation mode

Appendix: Qubes Naming and Color Conventions

This is just to give you some ideas. YMMV.

  • Names: codes you can add to qubes you make
    • wx : accesses the network via sys-whonix
    • app : app (uses template)
    • dvm : disposible vm
    • usb : accepts usb connections
    • vault : filestorage, no networking
    • blackhole : nothing comes out, no usb; used for repeat interaction with hostile media, esp reading pdfs
    • player : plays media (from usb drive)
  • Colors
    • red : clearnet (highest risk to anonymity and privacy)
    • orange : tor browser / whonix (reduced anonymity and privacy risk)
    • yellow : activity a
    • green : activity b
    • blue : usb (malware risk; usb 3 is blue)
    • purple : activity c
    • gray : vault (storage, transfers; files can come out)
    • black : blackhole (nothing comes out)

Appendix: Keybindings

  • Note: in the following scripts, be sure to replace <username> with your actual username.
  • Make the scripts executable: chmod +x <scriptname>

Starting Qubes and Apps

  • Type the following bash script template in dom0 (you can’t cut and paste it in for security reasons):
#!/bin/bash
# Identify *this* qube
Id=$(xdotool getwindowfocus)
Qube=$(xprop _QUBES_VMNAME -id $Id|cut -f2 -d \" )
# Check if this qube is dom0
if [[ $Qube == "_QUBES_VMNAME: not found." ]]
   then
       exit 1                                                # we don't launch from dom0
   else
       case $1 in
           # Launch apps inside *this* qube
           mousepad)       qvm-run $Qube mousepad         ;; # bind to <Shift-Ctrl-Alt-M>
           files-thunar)   qvm-run $Qube thunar           ;; # bind to <Shift-Ctrl-Alt-F>
           torbrowser)     qvm-run $Qube torbrowser       ;; # bind to <Shift-Ctrl-Alt-T>
           xfce4-terminal) qvm-run $Qube xfce4-terminal   ;; # bind to <Shift-Ctrl-Alt-X>
           qvm-backup)     qvm-run $Qube xfce4-terminal -x /home/user/SCRIPTBIN/qvm-backup  ;; # bind to <Shift-Ctrl-Alt-B>
           # Launch qubes if they are not already open: 'qvm-run <qube> <command>'
           anon)           qvm-run anon-whonix torbrowser ;; # bind to <Shift-Ctrl-Alt-A>
           work)           qvm-run work-wx xfce4-terminal ;; # bind to <Shift-Ctrl-Alt-W>
       esac
fi
  • Save it as run.sh in ~ in dom0
  • In Application Shortcuts, use commands like the following:
/home/<username>/run.sh "mousepad"           Shift+Ctrl+Alt+M
/home/<username>/run.sh "files-thunar"       Shift+Ctrl+Alt+F
/home/<username>/run.sh "qvm-backup"         Shift+Ctrl+Alt+B
etc
xfce4-terminal                               Shift+Ctrl+Alt+0  <- this will open a dom0 terminal

Controlling Screen Brightness

Something that matters a lot to me is dimming my multi-monitor setup. Here is a script that will do that.
First use xrandr in a dom0 terminal to check the IDs of your monitors (mine are HDMI-1 and HDMI-2).
Then type nano type this in:

#!/bin/bash

brightnessLevel=$(<"/home/<username>/brightnessLevel.txt")

[[ -z $brightnessLevel ]] && brightnessLevel=5

case $1 in
    +) brightnessLevel=$((brightnessLevel+1)) ;;
    -) brightnessLevel=$((brightnessLevel-1)) ;;
  '*') brightnessLevel=10 ;;
    /) brightnessLevel=2 ;;
    *) exit 1 ;; # failed attempt
esac

# We can't see anything if we set brightnessLevel too low, so set a reasonable lower limit (2)
[[ ${brightnessLevel} -le 1 ]]  && brightnessLevel=2
[[ ${brightnessLevel} -gt 10 ]] && brightnessLevel=10

if [[ ${brightnessLevel} -eq 10 ]]
   then
       xrandr --output HDMI-1 --brightness 1.0
       xrandr --output HDMI-2 --brightness 1.0
   else
       xrandr --output HDMI-1 --brightness 0."${brightnessLevel}"
       xrandr --output HDMI-2 --brightness 0."${brightnessLevel}"
fi

echo "${brightnessLevel}" > "/home/<username>/brightnessLevel.txt"

Save it as brightnessctrl.sh in ~.
Create its data file: touch brightnessLevel.txt
Set the following in Keyboard; Application Shoftcuts to activate it:

/home/<username>/brightnessctrl.sh '*'     Shift+Ctrl+Alt+KP *
/home/<username>/brightnessctrl.sh '/'     Shift+Ctrl+Alt+KP /
/home/<username>/brightnessctrl.sh '+'     Shift+Ctrl+Alt+KP +
/home/<username>/brightnessctrl.sh '-'     Shift+Ctrl+Alt+KP -

Backup Script

Since each qube has its own filesystem, one of the biggest challenges is managing data across many qubes. It is important for you to think about the important files in every qube, and put them somewhere you can find them. While QubesOS backsup qubes regularly, backing up user data is up to you, and is a bit more tricky. There are many backup solutions out there, but none of them are suited to Qubes.

To fill this gap, I offer the following simple script, which works in Whonix and delivers a backup file to backup-vault. YMMV.

#!/bin/bash

# @rideordie https://forums.whonix.org
# License: AGPL

# qvm-backup
# VERSION: 0.5b
# This backups essential files, like config files
# Sends to `backup-vault`, but any target qube do
# Optional software requirements: lz4jsoncat, yq

# NOTE: Sending files to another qube breaks isolation
# To maintain security, don't run anything in the backup-vault

# V A R I A B L E S   A N D   C O N S T A N T S

AutoConfirm="false"
Green="\033[92m"
Red="\033[91m"
Cyan="\033[36m"
Dim="\033[0;2;3m"
Reset="\033[0m"
Newline='
'

# F U N C T I O N S 

die() {
  # Need a way to pause exit of script called by keybinding
  # So users can read command output before window closes
  echo -en "  ${Dim}Press Any Key${Reset}"
  read Pause
  unset Pause
  exit $1
}

usage() {
     echo
     echo -e "  ${Cyan}qvm-backup: backs up important data to backup-vault${Reset}"
     echo "  USAGE: qvm-backup"
     echo 
     echo "    Will automatically setup if '~/BACKUP' dir is not present"
     echo "      Delete '~/BACKUP' to re-run setup"
     echo "    Edit 'include.txt' to select files you want to backup"
     echo "    Everything is automatic from there"
     echo "    Use the keybinding: <Shift-Ctrl-Alt-B>"
}

[[ -n $1 ]] && { usage; exit 0; }

confirm() {
    [[ $AutoConfirm == "true" ]] && { echo -e "  ${Cyan}${1} [y/N]?${Reset} Y"; return 0; }
    local Confirm
    echo -ne "  ${Cyan}${1} [y/N]? ${Reset}"
    read -sn 1 Confirm
    [[ ! $Confirm == $Newline ]] && echo
    case "${Confirm^^}" in
        Y) return 0 ;;
        *) return 1 ;;
    esac
    return 1
}

first_run() {
  # setup qvm-backup
  # NOTE: I clone git repositories into ~/GIT-REPOS, which is why I filter "GIT" (all caps) out with grep
  confirm "Setup qvm-backup" || die 0
  [[ ! -d ${BackupDir} ]] && mkdir "${BackupDir}"
  touch "${BackupDir}/include.txt"
  confirm "  Accept all changes" && AutoConfirm="true"
  confirm "  Add ~/Documents" && echo "/home/user/Documents/" >> "${BackupDir}/include.txt"
  confirm "  Add ~/Downloads" && echo "/home/user/Downloads/" >> "${BackupDir}/include.txt"
  confirm "  Add ~/Pictures" && echo "/home/user/Pictures/" >> "${BackupDir}/include.txt"
  confirm "  Add ~/Music" && echo "/home/user/Music/" >> "${BackupDir}/include.txt"
  confirm "  Add ~/Videos" && echo "/home/user/Videos/" >> "${BackupDir}/include.txt"
  confirm "  Add ~/Public" && echo "/home/user/Public/" >> "${BackupDir}/include.txt"
  confirm "  Search for config files" && find -maxdepth 4 \( -name "*config*" -o -name "*conf*" -o -name "*toml*" -o -name "*yaml*" -o -name "*yml*" -o -name "*txt*" -o -name "*json*" -o -name "*rc*" \) -print0 | xargs -0 -I {} file {} | grep "text" | awk '{ print $1}' | sed -e 's/.$//' -e 's/^\.\//\/home\/user\//g' | grep -v "GIT" >> "${BackupDir}/include.txt"
  confirm "  Add dotfiles (i.e., '.zshrc')" && find /home/user/ -maxdepth 1 -type f -name ".*" >> "${BackupDir}/include.txt"
  confirm "  Add dot dirs (i.e., '.tmux')" && find /home/user/ -maxdepth 1 -type d -name ".*" | grep -vE "cargo|local|cache|rust|tb|config" >> "${BackupDir}/include.txt"
  confirm "  Add fonts" && echo "/home/user/.local/share/fonts/" >> "${BackupDir}/include.txt"
  confirm "  Add TorBrowser profile data" && echo "/home/user/.tb/tor-browser/Browser/TorBrowser/Data/Browser/profile.default/places.sqlite" >> "${BackupDir}/include.txt"
  confirm "  Add ~/ALLCAPS folders" && find /home/user/ -maxdepth 1 -regex '.*[A-Z]+' | grep -v "GIT" >> "${BackupDir}/include.txt"
  confirm "  Add TorBrowser Downloads (Warning: size hazard)" && echo "/home/user/.tb/tor-browser/Browser/Downloads/" >> "${BackupDir}/include.txt"
  confirm "  Add executables (i.e., from go, rust, local-bin) (Warning: size hazard)" && find /home/user/ -maxdepth 3 -type d -name "bin" | grep -v GIT | xargs -I {} find {} | grep -v "bin$" >> "${BackupDir}/include.txt"
  confirm "  Add QubesIncoming (Warning: size hazard)" && echo "/home/user/QubesIncoming/" >> "${BackupDir}/include.txt"

  confirm "  Add command: backup Firefox Browser bookmarks (original)" && echo "add-firefox-bookmarks-original" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup Firefox Browser bookmarks (human-readable; install 'yq' and 'lz4jsoncat')" && { \
    [[ ! $(which yq) ]] && sudo apt-get -y install yq; \
    [[ ! $(which lz4jsoncat) ]] && sudo apt-get -y install lz4json; \
    "add-firefox-bookmarks-readable" >> "${BackupDir}/include.txt"; \
  }
  confirm "  Add command: backup Tor Browser bookmarks (original)" && echo "add-tor-bookmarks-original" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup Tor Browser bookmarks (human-readable; install 'yq' and 'lz4jsoncat')" && { \
    [[ ! $(which yq) ]] && sudo apt-get -y install yq; \
    [[ ! $(which lz4jsoncat) ]] && sudo apt-get -y install lz4json; \
    echo "add-tor-bookmarks-readable" >> "${BackupDir}/include.txt"; \
  }
  confirm "  Add command: backup list of exectuables in ~/.local/bin" && echo "add-executables-list" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup history of apt commands (installed packages)" && echo "add-apt-packages-list" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup snap list (installed packages)" && echo "add-snap-packages-list" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup flatpack list (installed packages)" && echo "add-flatpack-packages-list" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup pipx list (installed packages)" && echo "add-pipx-packages-list" >> "${BackupDir}/include.txt"
  confirm "  Add command: backup rust install list (installed packages and executables)" && echo "add-rust-packages-list" >> "${BackupDir}/include.txt"
  # sort and remove duplicates
  bat "${BackupDir}/include.txt"
  sort "${BackupDir}/include.txt" | uniq >> "${BackupDir}/include_uniq.txt"
  rm "${BackupDir}/include.txt"
  mv "${BackupDir}/include_uniq.txt" "${BackupDir}/include.txt"
#  bat "${BackupDir}/include.txt"
  echo -e "${Cyan}${BackupDir}/include.txt has been "
  echo -e "${Cyan}Remove or comment out any lines you don't need or don't understand"
  echo -e "${Cyan}Setup complete."
  die 0
}

# S E T U P

# BackupDir contains information about what you want backed up (persistent info)
BackupDir="/home/user/BACKUP"
[[ ! -d $BackupDir ]] && first_run
# BackupStage is where the backup archive is assembled (temp info)
BackupStage="/home/user/BACKUP-STAGE"
[[ ! -d $BackupStage ]] && mkdir $BackupStage
# NOTE: Due to ':', tar will choke on the file
# To access the backup file later, use the following command:
# $ tar --force-local -xvf <backup-file>
# Or open us using the graphical file browser
BackupDate="$(date +"%Y-%m-%d_%T")"
BackupFile="${BackupDate}_backup.tar"

# M A I N

# Estimate backup size
echo -ne "  ${Cyan}Estimating archive size...${Reset}"
FreeSpace=$(df --output=avail,target | grep --color=none /rw | awk '{ print $1 }')
while read Line; do
  [[ ! $Line =~ "#" && ! $Line =~ "add" ]] && lsd --total-size --size bytes --color never -AdLSF --blocks size,name "$Line" >> "${BackupDir}/${BackupDate}_size-estimate.txt"
done <"${BackupDir}/include.txt"
EstSize=$(cat "${BackupDir}/${BackupDate}_size-estimate.txt" | sort -r | awk '{ print $1}' | paste -sd+ - | bc )
# Check if there's enough space in /home/user/
if [[ ! $(echo "$FreeSpace - $EstSize" | bc | grep -v "\-" ) ]]; then
   echo
   echo -e "    ${Red}Insufficient disk space. Freespace: $(echo $FreeSpace | numfmt --to=iec); Backup: $(echo ${EstSize} | numfmt --to=iec)${Reset}"
   echo -e "    ${Red}Backup cancelled${Reset}"
   confirm "  Do you want to see the largest files and folders" && { \
       cat "${BackupDir}/${BackupDate}_size-estimate.txt" | sort -rn | awk '{ foo = $1 / 1024 / 1024; print foo "MB" "  " $2 }' | head -n 10 | grep -v "^0" | awk -v dim="$Dim" -v reset="$Reset" '{ print dim "      " $0 reset }'; \
       echo -e "    ${Cyan}Try commenting out or more of these lines in 'include.txt'${Reset}"; \
   }; \
   rm "${BackupDir}/${BackupDate}_size-estimate.txt"
   die 1
fi 
# If the backup is estimated to be especially large (1GB+), request confirmation
EstSize=$(echo "$EstSize" | numfmt --to=iec )
if [[ $EstSize =~ [GT] ]]
   then
       echo 
       confirm "  ${Red}Warning: ${Cyan}Size is ${Red}${EstSize// /}${Cyan}. Do you want to continue" || { \
           echo -e "    ${Red}Backup cancelled${Reset}"; \
           confirm "  Do you want to see the largest files and folders" && { \
               cat "${BackupDir}/${BackupDate}_size-estimate.txt" | sort -rn | awk '{ foo = $1 / 1024 / 1024; print foo "MB" "  " $2 }' | head -n 10 | grep -v "^0\."; \
               echo -e "    ${Dim}Try commenting out or more of these lines in 'include.txt'${Reset}"; \ 
           }; \
           rm "${BackupDir}/${BackupDate}_size-estimate.txt"; \
           die 0; \
       }
   else
       echo -e "${Green} OK${Reset}"
fi
rm "${BackupDir}/${BackupDate}_size-estimate.txt"

# Build backup
echo -e "  ${Cyan}Assembling backup...${Reset}"
tar -cf "${BackupStage}/${BackupFile}" ${BackupDir}/* 2>/dev/null 
# add contents of `includes.txt`
echo -e "  ${Cyan}Appending includes...${Reset}"
while read Line; do
  [[ ! $Line =~ "#" ]] && echo -e "  ${Dim}  ${Line}${Reset}"
  [[ ! $Line =~ "#" && ! $Line =~ "^add" ]] && tar --append -f "${BackupStage}/${BackupFile}" "$Line" 2>/dev/null
  [[ $Line =~ "add-tor-bookmarks-original" && -d /home/user/.tb/ ]] && cp "$(ls -1t /home/user/.tb/tor-browser/Browser/TorBrowser/Data/Browser/profile.default/bookmarkbackups/*.jsonlz4 | head -n 1)" "${BackupDir}/"
  [[ $Line =~ "add-tor-bookmarks-readable" && -d /home/user/.tb/ ]] && lz4jsoncat "$(ls -1t /home/user/.tb/tor-browser/Browser/TorBrowser/Data/Browser/profile.default/bookmarkbackups/*.jsonlz4 | head -n 1)" | yq -y | grep -E "title|uri" | grep -v "iconUri" | sed -e 's/\ \ /\ /g' -e 's/^\ \ \ \ //g' -e 's/title:/=/g' -e 's/uri:/\ \ >/g' -Ee 's/\=\ ([A-Z0-9 -]*)$/\/ \1/g' > "${BackupDir}/${BackupDate}_tor-bookmarks.txt"
  [[ $Line =~ "add-firefox-bookmarks-original" && -d /home/user/.mozilla/ ]] && cp "$(find /home/user/.mozilla -type f -name "bookmarks*jsonlz4" | sort -r | head -n 1)" "${BackupDir}/"
  [[ $Line =~ "add-firefox-bookmarks-readable" && -d /home/user/.mozilla/ ]] && lz4jsoncat "$(find /home/user/.mozilla -type f -name "bookmarks*jsonlz4" | sort -r | head -n 1)" | yq -y | grep -E "title|uri" | grep -v "iconUri" | sed -e 's/\ \ /\ /g' -e 's/^\ \ \ \ //g' -e 's/title:/=/g' -e 's/uri:/\ \ >/g' -Ee 's/\=\ ([A-Z0-9 -]*)$/\/ \1/g' > "${BackupDir}/${BackupDate}_firefox-bookmarks.txt"

  [[ $Line =~ "add-executables-list"   ]] && find /home/user/ -maxdepth 3 -type d -name "bin" | grep -v GIT | xargs -I {} find {} | grep -v "bin$" > "${BackupDir}/${BackupDate}_executables-list.txt"
      
  if [[ $Line =~ "add-apt-packages-list" ]] 
     then 
         grep Commandline /var/log/apt/history.log | grep -v Dpkg > "${BackupDir}/${BackupDate}_apt-history.txt"
         if [[ -f "/var/log/apt/history.log.1.gz" ]]
            then
                for a in /var/log/apt/history.log.*.gz; do 
                    gunzip -c $a | grep Commandline | grep -v Dpkg; 
                done >> ${BackupDir}/${BackupDate}_apt-history.txt
         fi
  fi
  [[ $Line =~ "add-pipx-packages-list"     ]] && pipx list            2>/dev/null > "${BackupDir}/${BackupDate}_pipx-list.txt"
  [[ $Line =~ "add-snap-packages-list"     ]] && snap list            2>/dev/null > "${BackupDir}/${BackupDate}_snap-list.txt"
  [[ $Line =~ "add-flatpack-packages-list" ]] && flatpack list        2>/dev/null > "${BackupDir}/${BackupDate}_flatpack-list.txt"
  [[ $Line =~ "add-docker-packages-list"   ]] && flatpack list        2>/dev/null > "${BackupDir}/${BackupDate}_docker-list.txt"
  [[ $Line =~ "add-rust-packages-list"     ]] && cargo install --list 2>/dev/null > "${BackupDir}/${BackupDate}_rust-list.txt"
done <"${BackupDir}/include.txt"
# gzip the archive
echo -e "  ${Cyan}Zipping archive...${Reset}"
gzip "${BackupStage}/${BackupFile}" 
echo -e "  ${Cyan}Transferring archive...${Reset}"
# move backup file to backup-vault (requires interaction)
qvm-move-to-vm backup-vault "${BackupStage}/${BackupFile}.gz"
echo -e "  ${Cyan}Cleanup...${Reset}"
# remove intermediate files in $BackupDir
# But only if the variable $BackupDir is not empty (safer rm)
[[ -n $BackupDir && ! ${BackupDir// /} == "/" ]] && rm ${BackupDir}/${BackupDate}_*
[[ -n $BackupDir && ! ${BackupDir// /} == "/" ]] && rm ${BackupDir}/*.jsonlz4
# Remove ${BackupStage} since backup has been moved out
rmdir "${BackupStage}" 2>/dev/null || { \
    echo -e "    ${Red}Warning: ${Cyan}${BackupStage} is not empty${Reset}"; \
    echo -e "    ${Cyan}Check if '${BackupFile}.gz' was transferred${Reset}"; \
    }
echo -e "  ${Cyan}Done.${Reset}"
die 1

Save it in ~/SCRIPTBIN, or wherever you want, but adjust the section on keybindings to reflect the change. You can now backup essential information with <Shift-Ctrl-Alt-B> and a menu interaction.

1 Like
2

A post was split to a new topic: Discourse Footnote