upstream bind-directories functionality to Qubes

phabricator-migrator · February 19, 2024, 5:47pm

Information

ID: 414
PHID: PHID-TASK-djsva3n7qdfaip6rf3im
Author: Patrick
Status at Migration Time: resolved
Priority at Migration Time: High

Description

(As discussed in don't really boot TemplateVM, only chroot into those · Issue #1306 · QubesOS/qubes-issues · GitHub)

Upstream https://github.com/Whonix/qubes-whonix/blob/master/usr/lib/qubes-whonix/bind-directories to Qubes.

Comments

Patrick

2015-10-14 08:35:53 UTC

marmarek

2015-10-14 08:52:56 UTC

Patrick

2015-10-14 10:28:54 UTC

Patrick

2015-10-15 16:08:25 UTC

Patrick

2015-10-19 21:46:16 UTC

marmarek

2015-10-19 21:49:21 UTC

Patrick

2015-10-19 22:24:31 UTC

marmarek (Marek Marczykowski-Górecki):

I think both.

Ok.

Somehow offtopic: for per-VM configuration would you prefer /rw/config
(Qubes-specific path), or /usr/local/etc (somehow more standard)?
Maybe those two should be merged (using symlink)?

Hm. This is an important decision. I am always having in mind and
hoping, that some day, packages such as qubes-core-agent etc. will be
uploaded to the official Debian archive. [So distributions such as Tails
could simply install it.]

Neither. /rw/ clearly violates FHS. /usr/local/etc/ looks even more
strange. Debian policy forbids packages to write into local.

What about /home/config/? Hehe. No. Same.

What about something like /etc/qubes/rw/?

Compatibility symlink [or better, if possible, bind-mount] is a good idea.

Let’s also ask at least another person. @bnvk

Also imho low priority, but okay to change.

marmarek

2015-10-19 22:35:57 UTC

Patrick

2015-11-12 20:30:33 UTC

! In T414#7005, @Patrick wrote:
Where should this feature be configureable?

a) from TemplateVM only → /etc/qubes-bind-dirs.d/, or

b) from TemplateBasedVM only → /rw/config/qubes-bind-dirs.d/, or

c) both, a) and b)?

! In T414#7006, @marmarek wrote:
I think both.

Wrong/non-existing entries should not make the script fail, right?

I guess, parse /etc/qubes-bind-dirs.d/ first (lower priority), parse /rw/config/qubes-bind-dirs.d/afterwards (higher priority)?

Should higher named/priority files (example: 50_user) in qubes-bind-dirs.d be capable to disable items by lower priority files (example: 30_qubes)?

I assume yes for now.

This is my far from prototype test script for now.
#!/bin/bash

#set -x
set -e

bind_directories+=('test file')
bind_directories+=('another one')
bind_directories+=('bar test')

delete() {
   bind_directories=( "${bind_directories[@]/'another one'}" )
}

#delete

array_length="${#bind_directories[@]}"
i=-1

while true; do
   i=$(( i + 1))

   if [ "$i" -ge "$array_length" ]; then
      break
   fi

   if [ "${bind_directories[$i]}" = "" ]; then
      ## Fix empty or removed ones.
      continue
   fi
   echo "${bind_directories[$i]}"

done
In the qubes-bind-dirs.d config folders, packages or users could drop bash snippets in the following form.

For adding:
bind_directories+=('test file')
bind_directories+=('another one')
bind_directories+=('bar test')
Or the more difficult case of removal:
bind_directories=( "${bind_directories[@]/'another one'}" )

marmarek

2015-11-12 20:52:20 UTC

Wrong/non-existing entries should not make the script fail, right?

Right. Maybe entries like /etc/something → /rw/config/something
should try to create /rw/config/something? Or even initialize with
original /etc/something content? That would make per-VM configuration
of virtually any service really easy.

I guess, parse /etc/qubes-bind-dirs.d/ first (lower priority), parse /rw/config/qubes-bind-dirs.d/afterwards (higher priority)?

Yes. Or even follow systemd practice and have
/usr/lib/qubes-bind-dirs.d with lowest priority? Not sure about that.

Should higher named/priority files (example: 50_user) in qubes-bind-dirs.d be capable to disable items by lower priority files (example: 30_qubes)?

I assume yes for now.

I think 30_qubes in higher priority dir should override 30_qubes in
lower priority dir.

If we’d have /usr/lib/qubes-bind-dirs.d (or simply
/usr/lib/qubes-bind-dirs?), then user may override it with file in
either /etc (Template wide), or /rw/config (only one VM).

Patrick

2015-11-12 21:09:40 UTC

marmarek (Marek Marczykowski-Górecki):

marmarek added a comment.

Wrong/non-existing entries should not make the script fail, right?

Right. Maybe entries like /etc/something → /rw/config/something
should try to create /rw/config/something?

Sorry, I am not getting it.

You mean it should be a simpler config file format such as
/etc/something1
/etc/something2
/etc/something3
?

Not using the variable+= syntax?

Not sure if I could implement that. Perhaps. Then the config parsing
code would be more complicated than a simple source.

Or even initialize with
original /etc/something content? That would make per-VM configuration
of virtually any service really easy.

I am not following either.

I guess, parse /etc/qubes-bind-dirs.d/ first (lower priority), parse /rw/config/qubes-bind-dirs.d/afterwards (higher priority)?

Yes. Or even follow systemd practice and have
/usr/lib/qubes-bind-dirs.d with lowest priority? Not sure about that.

Can do. Only minimally more code. (One more entry in a for loop.)

Should higher named/priority files (example: 50_user) in qubes-bind-dirs.d be capable to disable items by lower priority files (example: 30_qubes)?

I assume yes for now.

I think 30_qubes in higher priority dir should override 30_qubes in
lower priority dir.

Yes. Higher priority dirs always overwrite lower priority dirs
independently of lexical order. (That’s seemingly happening mostly
nowadays.)

If we’d have /usr/lib/qubes-bind-dirs.d (or simply
/usr/lib/qubes-bind-dirs?),

With ‘.d’ is better.

then user may override it with file in
either /etc (Template wide), or /rw/config (only one VM).

Sounds good.

marmarek

2015-11-12 21:39:59 UTC

I was thinking about simple config format, very similar to the current list:

/rw/config/something1:/etc/something1
/rw/config/something2:/etc/something2
/rw/config/something3:/var/lib/something3

Then load them with something like:

bind_dirs=""
for f in $(ls /usr/lib/qubes-bind-dirs.d /etc/qubes-bind-dirs.d /rw/config/qubes-bind-dirs.d | sort | uniq); do
  if [ -r "/rw/config/qubes-bind-dirs.d/$f" ]; then
    bind_dirs+=$(grep -v '^#' "/rw/config/qubes-bind-dirs.d/$f")
  elif [ -r "/etc/qubes-bind-dirs.d/$f" ]; then
    bind_dirs+=$(grep -v '^#' "/etc/qubes-bind-dirs.d/$f")
  elif [ -r "/usr/lib/qubes-bind-dirs.d/$f" ]; then
    bind_dirs+=$(grep -v '^#' "/usr/lib/qubes-bind-dirs.d/$f")
  else
    echo "WAT?!"
    exit 1
  fi
done

# Then process loaded entries:
for entry in ${bind_dirs}; do
 ...
done

Maybe better make ${bind_dirs} an array to better handle spaces. You’re much better bash wizard than me, you’ll do it right

Patrick

2015-11-30 18:12:09 UTC

This is what I have for now.

https://github.com/adrelanos/qubes-core-agent-linux/tree/bind-dirs

https://github.com/adrelanos/qubes-core-agent-linux/commit/ad99a0ae75eb048b309d435601bba81d93af19ae

no longer requires rsync

now works also for files, not just folders

sync removed, should not be required

sorted out user access rights issues

A detailed review is not required yet. It’s not fully tested yet. Only a glimpse would be nice.

However, I am wondering about the following…

What to do about symlinks? Copying them as symlink is possible and implemented. But mount fails. Example.
mount --bind /rw/bind-dirs/etc/hosts /etc/hosts
mount: special device /rw/bind-dirs/etc/hosts does not exist
So we could just skip mounting for symlinks. The symlinks that are copied to /rw/bind-dirs do link the the proper files in the system.

But then modifications to for example /etc/some-sym-link would not end up in /rw/bind-dirs.

Maybe bind-dirs should delete the old symlink /etc/some-sym-link. And create a new symlink for its place to /rw/bind-dirs/etc/some-sym-link?

The same in other words.

Originalls /etc/hosts is a symlink to /etc/hosts.anondist.
Now the user adds /etc/hosts to bind-dirs configuration.
bind-dirs will
“unlink /etc/hosts”
“ln -s /etc/hosts.anondist /rw/bind-dirs/etc/hosts”
“ln -s /rw/bind-dirs/etc/hosts /etc/hosts.”

(Cannot mv symlinks - would not work for relative symlinks.)

A two layered symlink. But I guess that could cause some issues.

Another option would be follow the symlink. Copy the file it links to to /rw/bind-dirs. Then mount it as usual. However, I don’t know if that could also cause issues. Perhaps if any scripts check if they got the symlink or otherwise. Probably less issues than above.

What about file system objects added to bind-dirs list, that do not exist? Fail closed, abort, exit error? Fail silent, open? Write to stderr, continue, fail open?

What about other file system objects such as block, character, fifo, special, device, … (?)?

Patrick

2015-11-30 21:41:40 UTC

Posting the old notes by @nrgaway here.

'bind-directories' will automatically bind important Whonix configuration
directories within root image filesystem to the users /rw/ filesystem so the
changes will persist upon reboot eliminating the need to have Whonix be a
standalone VM.

The first time an AppVM is started the original content from the root image
are copied to the users /rw directory (one-time) to make sure any changes
that have already been made in template are reflected in AppVM.

Any further changes, like enabling 'Tor' are stored directly in the 'rw'
filesystem.  For example '/etc/tor' is bound to '/rw/Whonix/etc/tor'.

If a user then makes any changes in the TemplateVM, those changes will not
be reflected in the AppVM after the initial bind which may cause confusion
if a user is attempting to make configuration changes in the TemplateVM.

Possible solutions:
  - track original date / shasum of files when originally copying defaults to /rw
  - store that in /rw as .bind-dirs/filename ?
  - before binding; check for changes; update rw if there are changes; maybe
    just using file date would work?

Patrick

2015-11-30 22:28:02 UTC

! In T414#7386, @Patrick wrote:
Posting the old notes by @nrgaway here.
'bind-directories' will automatically bind important Whonix configuration
directories within root image filesystem to the users /rw/ filesystem so the
changes will persist upon reboot eliminating the need to have Whonix be a
standalone VM.

The first time an AppVM is started the original content from the root image
are copied to the users /rw directory (one-time) to make sure any changes
that have already been made in template are reflected in AppVM.

Any further changes, like enabling 'Tor' are stored directly in the 'rw'
filesystem.  For example '/etc/tor' is bound to '/rw/Whonix/etc/tor'.

If a user then makes any changes in the TemplateVM, those changes will not
be reflected in the AppVM after the initial bind which may cause confusion
if a user is attempting to make configuration changes in the TemplateVM.
Right. I recognize the issue. We’re adding another layer of complexity. Not great. Likely to cause confusion. However, I don’t think there can be a solution to this issue. What we could consider would be something more drastic. [1]
Possible solutions:
  - track original date / shasum of files when originally copying defaults to /rw
  - store that in /rw as .bind-dirs/filename ?
  - before binding; check for changes; update rw if there are changes; maybe
    just using file date would work?
Okay, let’s say you somehow found a modification. Was it done by the user or by the package manager? To my knowledge, there is no way to reliably distinguish this. But let’s put that aside for now. Let’s say the user made a modification. Now, we might have two modified versions. For example a modified /etc/tor/torrc in the Whonix-Gateway TemplateBasedProxyVM, and a different modified /etc/tor/torrc inside the whonix-gw TemplateVM. Which changes we keep, which ones we discard? Do we [silently] discard settings that were made in the TemplateBasedVM? Sounds confusing. Ask the user to merge them? Probably not. Do we try something as crazy as automatically merging them? Probably not. I don’ think this is solvable, but I am happy to hear ideas.

What however might be doable, would be the following.

somehow track the file hashes of the files from non-persistent

track the file hashes of rw once we initially copy them over from non-persistent to rw

notice changes in the non-persistent folders (after TemplateVM upgrades)

now if the file in rw is still unmodified, we could copy over the modified version from non-persistent to rw to update the stale one on rw

But I think instead of making things easier, it would only add even more complexity and confusion.

[1] two different drastic ideas that do not rely on bind-directories

Making Whonix-Gateway ProxyVM a standalone VM by default. Not great for backups. Would waste space using Qubes usual backups. But while Whonix-Gateway backups are useful - keeping Tor entry guards; keeping Tor/Firewall config, perhaps keeping misc other config - is not super important. And also not well documented/encouraged at the moment.

Configure/hack Tor running as user user rather than system user debian-tor. Then Tor persistent data would be stored in the user’s home folder. Other stuff such as Whonix Firewall settings are much less problematic and could also be stored in the user’s home folder instead.

Maybe these are some interesting future directions. If this sounds interesting, we can move that discussion into a separate ticket.

(Even in case we decided to not use bind-directories by default it would be useful to finish this tool even if it will not be used by default.)

marmarek

2015-12-06 01:18:39 UTC

[1] two different drastic ideas that do not rely on bind-directories

Making Whonix-Gateway ProxyVM a standalone VM by default. Not great for backups. Would waste space using Qubes usual backups. But while Whonix-Gateway backups are useful - keeping Tor entry guards https://www.torproject.org/docs/faq.html.en#EntryGuards; keeping Tor/Firewall config, perhaps keeping misc other config - is not super important. And also not well documented/encouraged at the moment.

It isn’t only about backups. It’s also about disk space on actual
system.

Configure/hack Tor running as user user rather than system user debian-tor. Then Tor persistent data would be stored in the user’s home folder. Other stuff such as Whonix Firewall settings are much less problematic and could also be stored in the user’s home folder instead.

This doesn’t solve anything. Still that config will initialized from
somewhere, then stored in home directory and not updated (from initial
source) ever. So basically you have the same “problem” as with bind
directory, but moved to home.

I think the one time synchronization (at the first VM start time) is the way
to go.

Patrick

2015-12-06 12:29:22 UTC

Patrick

2015-12-06 17:01:32 UTC

Patrick

2015-12-18 00:00:45 UTC

Patrick

2015-12-18 00:20:50 UTC

Patrick

2015-12-25 12:40:03 UTC

https://github.com/marmarek/qubes-core-agent-linux/pull/58

Optional debug script that may be useful.

#!/bin/bash
set -x
set -e

umount /etc/testdir || true
umount /etc/testsymlink || true
rm -r /etc/testsymlink || true
rm -r /etc/testdir || true
mkdir /etc/testdir
echo test > /etc/testdir/a
ln -s /etc/testdir /etc/testsymlink
ls -la /etc/testsymlink
ls -la /etc/testsymlink/

sleep 1

sudo ./misc/bind-dirs 1
sudo rm -r /rw/bind-dirs
sudo ./misc/bind-dirs
ls -la /etc/hosts
ls -la /rw/bind-dirs/etc/hosts

Patrick

2016-04-21 21:16:05 UTC