My hypothesis is, that somehow through the Qubes menu there is no terminal attached?
Might be a bug in Qubes. To falsify/confirm, could you create a test script to the menu please that check whether running in a terminal or not? Perhaps a loop that waits until terminal gets available? This would help (check for stdin):
I could help with that test script if necessary.
But perhaps Whonix scripts that use terminal input should check if they’re really running in a terminal and otherwise wait for the terminal forever?
About 1.) Probably similar to 2.) [Using dialog while no terminal is attached.]
[quote=“Patrick, post:77, topic:512”][quote]1. Whonix Repository: Tries to start and exits
2. Configuration Manager (whonixsetup): It won’t show and will actually shutdown (power off) the VM[/quote]
My hypothesis is, that somehow through the Qubes menu there is no terminal attached?[/quote]
I think you are right. Qubes make a call to the VM to start x-window programs. If you want an actual terminal you call the gnome-terminal application.
Might be a bug in Qubes. To falsify/confirm, could you create a test script to the menu please that check whether running in a terminal or not? Perhaps a loop that waits until terminal gets available? This would help (check for stdin):
http://stackoverflow.com/questions/9863508/how-do-i-detect-whether-stderr-is-connected-to-a-terminal-from-within-a-script
I could help with that test script if necessary.
But perhaps Whonix scripts that use terminal input should check if they’re really running in a terminal and otherwise wait for the terminal forever?
About 1.) Probably similar to 2.) [Using dialog while no terminal is attached.]
I will look into the script. Maybe a wrapper can be designed to start those types of apps within gnome-terminal.
We can work around it in Whonix, but I think it should be reported against Qubes as well.
For the report against Qubes, a small test script
/usr/share/applications/teststdin.desktop → /usr/bin/teststdin
that reads from stdin (which will fail as I expect) or uses dialog (which I am more surely expect to fail).
[quote]1. Whonix Repository: Tries to start and exits
2. Configuration Manager (whonixsetup): It won’t show and will actually shutdown (power off) the VM[/quote]
My hypothesis is, that somehow through the Qubes menu there is no terminal attached?[/quote]
I think you are right. Qubes make a call to the VM to start x-window programs. If you want an actual terminal you call the gnome-terminal application.[/quote]
Indeed Qubes currently ignores Terminal=true option in desktop file, only Exec= line is handled. I’ll fix it shortly.
I was about to let you know I mentioned the issue to Marek but I guess he got to it first
Now since I have everything working I can re-address the IP address issue. Since there are not many files to have to search and replace with an IP address and that IP address will not (most likely) change once it is set I am satisfied by using this procedure. I am able to do the search and replace before the network comes up and is quick.
In order for this to be reliable, I will also need to re-activate the search after an update as you previously mentioned. What file should my script be called from (I call ‘/usr/lib/whonix/setup-ip’ which setups up eth1 if needed, makes sure hosts hostname and resolv.conf are set properly, calls replace-ips in same directory, then restarts tor if ip changed and then reloads whonix_firewall).
I propose as part of the Whonix build process that a new package gets generated that would list the file names that need to be searched and replaced and that file be part of a package which would then end up on the users computer when they do an update. That way if any new module needs to be parsed, my script will know about it since it would take forever to scan the entire contents of the drive and could change unrelated files. This file could look like this (can be same file for both gateway / ws since if file does not exist it does not get replaced)
I also suggest in the same package we keep a list of ip address that may need to be searched for in case they ever change mapped to an interface. The interface name does not have to be eth0 / eth1 since that is not descriptive enough for me to parse if things change, so maybe follow your firewall interface names of INT_IF INT_TIF, etc
Any comments or suggestions on this proposed format?
Also, I was wondering if you would have any advice of using Travis CI. I think it would be a good idea to set something like that up. With you code on github, did you need to set up your own private version of Travis or are you able to use a public server? I have never used CI before so any helpful starting point would be appreciated. I have a Ubuntu server I can run it on locally if needed.
We can work around it in Whonix, but I think it should be reported against Qubes as well.
For the report against Qubes, a small test script
/usr/share/applications/teststdin.desktop → /usr/bin/teststdin
that reads from stdin (which will fail as I expect) or uses dialog (which I am more surely expect to fail).[/quote]
As a stop-gap measure I just linked dialog to gdialog and that seems to be able to display the dialogs nicely. kdialog gave me empty box when running from shell (lots of Error: BadDrawable messages). Maybe it would be nice to have that set as an environment variable?
We can work around it in Whonix, but I think it should be reported against Qubes as well.
For the report against Qubes, a small test script
/usr/share/applications/teststdin.desktop → /usr/bin/teststdin
that reads from stdin (which will fail as I expect) or uses dialog (which I am more surely expect to fail).
[/quote]
As a stop-gap measure I just linked dialog to gdialog and that seems to be able to display the dialogs nicely. kdialog gave me empty box when running from shell (lots of Error: BadDrawable messages). Maybe it would be nice to have that set as an environment variable?
$DIALOG --title " My first dialog" --clear
–yesno “Hello , this is my first dialog program” 10 30
[/code][/quote]
I like that idea independently from the Qubes issue!
Using “$DIALOG” instead of “dialog” seems like a good idea. Then we can add at the beginning of the script some logic on which to prefer.
Perhaps depending on which is installed? Or just depend on it and assume it’s installed?
Somehow auto detect X and prefer “gdialog” then? Otherwise fall back to “dialog”?
And let an environment variable overrule the auto detection code?
You’ll probably want to turn “/sbin/poweroff” to “echo /sbin/poweroff” while development.
We’d have to run “kdesudo whonixsetup” (because “sudo” won’t work when X is involved). But we likely find a solution for that.
whonixsetup’s disclaimer is too big. The buttons are outside the visible area. I wanted to split them into more screens some day anyway (to make them work in the less-lines-by-default gnome-default-terminal).
That way we might be able to avoid a separate whonix-qubes-network-conf or even whonix-qubes-(gw|ws)-network-conf package.
[hr]
If we were going for “perfect performance”, we should create a whonix-gw-ip-files.list as well as a separate whonix-ws-ip-files.list file. But I propose we go for just one shared file for simplicity. A single whonix-ip-files.list file. Whonix-Gateway might run an obsolete “if [ -f /home/user/.torchat/torchat.ini ]” and skip that file, but “if [ -f … ]” is really fast. So I don’t think it will be that many files that it matters. (Also generating gw/ws specific ip list files would be much more error prone, because the script that generates them would need some understanding which package gets installed on either/and/or gw/ws.) I guess the performance loss will be absolutely negligible but the code much simpler.
[hr]
No idea where this fits best. Perhaps we create a new whonix-shared-network-conf package where we put all that functionality?
usr/share/whonix-shared-network-conf/whonix-gateway-ip (Contains GW_INT_IF:10.152.152.10) (Whonix developers must manually keep that file current. Not hard, because should we ever [I hope not!] change Whonix’s internal IPs again, we need to search and replace the whole Whonix source folder anyhow.)
usr/lib/whonix-shared-network-conf/ip-file-list-gen [–build-time]
– Searches through ./packages for IPs. Supposed to be run at build time.
– Creates a file /usr/share/whonix-shared-network-conf/ip-file-list. (Contains /home/user/.torchat/torchat.ini and so forth.)
– That file should be deterministic. (Probably the default and hard to make it non-deterministic.)
– Do we want to make that file versioned? Probably not required. Version number of the package suffices.
usr/lib/whonix-shared-network-conf/ip-file-list-gen --run-time
– Creates a file /var/cache/whonix-shared-network-conf/ip-file-list (OPTIONAL. You decide. [/var/cache or /var/lib. Needs very little research.] Not sure it’s really neccessary.)
usr/lib/whonix-shared-network-conf/ip-changer (If not Qubes → exit 0. Otherwise use /usr/(share|cache)/whonix-shared-network-conf/ip-file-list to do the search and replace magic.)
etc/[…] hooks that all run /usr/lib/whonix-shared-network-conf/ip-changer
(ip-file-list-gen and other names are not final. If you have better proposals, sure thing! Goes for the whole proposal!)
[hr]
As for “part of the Whonix build process”.
I am fine with running packages/whonix-shared-network-conf/usr/lib/whonix-shared-network-conf/ip-file-list-gen somewhere during the build process.
It would regenerate that file.
Ideally there are no changes.
Then it won’t matter.
But please no autogenerated files that are auto committed to git.
When that file changes, if Whonix maintainers forgot to manually run it, the build will break by default due to uncommitted git changes. In that case a Whonix maintainer will manually run ip-file-list-gen and commit the changes to git.
This will probably not happen often anyhow. So more autogeneration autocommitting than that would probably not be required.
As a stop-gap measure I just linked dialog to gdialog and that seems to be able to display the dialogs nicely. kdialog gave me empty box when running from shell (lots of Error: BadDrawable messages). Maybe it would be nice to have that set as an environment variable?
You’ll probably want to turn “/sbin/poweroff” to “echo /sbin/poweroff” while development.
We’d have to run “kdesudo whonixsetup” (because “sudo” won’t work when X is involved). But we likely find a solution for that.
whonixsetup’s disclaimer is too big. The buttons are outside the visible area. I wanted to split them into more screens some day anyway (to make them work in the less-lines-by-default gnome-default-terminal).
Would you like to push a branch, patch or commit?[/quote]
Marek has already push a commit I can try out tomorrow which I hope will help initially with those applications. In the mean time linking to gdialog seems to work. I was having an issue though where I need to wait until xdg applications were auto started to be able to use the Connection Manager; it was taking 45 seconds. I am trying to re-integrate the Connection Manager (whonixsetup) back in with tor disabled by default until users see the disclaimer and choose how to install.
I will be able to see how Marek commit may help tomorrow.
Once I have the build working Marek suggested I integrate files that may need updating into the main Whonix repo so they can be updated. There are only a handful so I will create a package for them. Mostly they deal with setting up the network and search and replace functions.
I am starting to remove any hacks needed out in favour of updating either Qubes or Whonix so they won’t be needed so at some point I will need to create a branch. I have not done it yet since I am having to change the logic often when a new issue comes up.
I think the last major item will be to get Whonix to be able to update from a template which has no network access by default and has to use a proxy when an update is requested. I was looking into this a bit today as well. I am thinking of loading different firewall rules for the template that would only allow access from apt-get and force tor to use the Qubes proxy (if that’s possible) for its outgoing connection to be sure that updates are made over Tor.
In Qubes the networking is done a bit backwards. I am responsible for the code to bring up the networks, or at least the internal interface. Currently I have a udev rules that will add the internal interface, replace the ips, protect hosts resolv.conf and hostname and start firewall. These will be some thing that will need to be updatable for bug fixes and would be a included in a package.
I see needing hooks after updating and maybe before whonixcheck is done and before time sync. I should sort this one out in next few days. The reason whonix was giving PID not found errors cause I was resetting tor after ip address have been changed, and it was looking at old IP I think. And if I disabled tor I got messages since it was expecting it to be running (all because I could not tun setup at an early stage).
My work-around was to disable whonixcheck and sdwdate and re-enable them once I was able to wait the 45 seconds to run setup. So this will hopefully change now with Marek’s commit… Tomorrow will tell
I might have said this before. Dunno. We can rely on the output of "sudo virt-what" and then "do-some-stuff if Qubes detected, otherwise exit 0".
That way we might be able to avoid a separate whonix-qubes-network-conf or even whonix-qubes-(gw|ws)-network-conf package.
Nice. Too easy!
If we were going for "perfect performance", we should create a whonix-gw-ip-files.list as well as a separate whonix-ws-ip-files.list file. But I propose we go for just one shared file for simplicity. A single whonix-ip-files.list file. Whonix-Gateway might run an obsolete "if [ -f /home/user/.torchat/torchat.ini ]" and skip that file, but "if [ -f ... ]" is really fast. So I don't think it will be that many files that it matters. (Also generating gw/ws specific ip list files would be much more error prone, because the script that generates them would need some understanding which package gets installed on either/and/or gw/ws.) I guess the performance loss will be absolutely negligible but the code much simpler.
[hr]
No idea where this fits best. Perhaps we create a new whonix-shared-network-conf package where we put all that functionality?
usr/share/whonix-shared-network-conf/whonix-gateway-ip (Contains GW_INT_IF:10.152.152.10) (Whonix developers must manually keep that file current. Not hard, because should we ever [I hope not!] change Whonix’s internal IPs again, we need to search and replace the whole Whonix source folder anyhow.)
usr/lib/whonix-shared-network-conf/ip-file-list-gen [–build-time]
– Searches through ./packages for IPs. Supposed to be run at build time.
– Creates a file /usr/share/whonix-shared-network-conf/ip-file-list. (Contains /home/user/.torchat/torchat.ini and so forth.)
– That file should be deterministic. (Probably the default and hard to make it non-deterministic.)
– Do we want to make that file versioned? Probably not required. Version number of the package suffices.
usr/lib/whonix-shared-network-conf/ip-file-list-gen --run-time
– Creates a file /var/cache/whonix-shared-network-conf/ip-file-list (OPTIONAL. You decide. [/var/cache or /var/lib. Needs very little research.] Not sure it’s really neccessary.)
usr/lib/whonix-shared-network-conf/ip-changer (If not Qubes → exit 0. Otherwise use /usr/(share|cache)/whonix-shared-network-conf/ip-file-list to do the search and replace magic.)
etc/[…] hooks that all run /usr/lib/whonix-shared-network-conf/ip-changer
(ip-file-list-gen and other names are not final. If you have better proposals, sure thing! Goes for the whole proposal!)
This will take some time to consider. One other consideration I have to work against is that Qubes uses base templates as default, where when an AppVM is run, it uses a copy of the original which is destroyed after the session. So that means anything that in not written and saved in /home/user or /rw (which is also /usr/local) will be lost. I have not taken much time yet to think it though on the best way to store configuration files. Marek suggested as a starting point to link them within the /rw directory somewhere and make sure they get populated on first run.
I’m thinking there could be a master configuration file in /etc/default that would specify where any user writable configuration files should go. Now they seem to be in many places (/root has some so does /home/user, /var/lib/whonix[.d] and the user configurable options within /etc, etc/tor, etc). Typically I would consider a master folder with sub-folders so only one directory would need to be referenced as the base. If the /srv directory was not read only, that’s where I would suggest the go.
The AppVM is updated by updating the master template. So when the master is updated all the AppVM’s that use it also automatically get updated.
I am also still on the encryption kick, but I am waiting for the next release of Qubes before I decide how to implement this. Initially it will start with home directory and /rw (home dir is mounted from /rw).
As for "part of the Whonix build process".
- I am fine with running packages/whonix-shared-network-conf/usr/lib/whonix-shared-network-conf/ip-file-list-gen somewhere during the build process.
- It would regenerate that file.
- Ideally there are no changes.
- Then it won't matter.
- But please no autogenerated files that are auto committed to git.
- When that file changes, if Whonix maintainers forgot to manually run it, the build will break by default due to uncommitted git changes. In that case a Whonix maintainer will manually run ip-file-list-gen and commit the changes to git.
- This will probably not happen often anyhow. So more autogeneration autocommitting than that would probably not be required.
[quote=“Patrick, post:89, topic:512”][quote author=nrgaway link=topic=537.msg5039#msg5039 date=1414478074]
I think the last major item will be to get Whonix to be able to update from a template which has no network access by default and has to use a proxy when an update is requested.
[/quote]
Does this help?
The answer to that question is quite vague. It needs more detail on how to disable yum-proxy, etc.
The question though seems to relate to a regular AppVM template that wants to use Tor for its updates. While I could implement this to be able to update Whonix-Gateway template though Tor that would be another netvm would need to be already setup that is running the Qubes Tor package.
What I want to do is to figure out how to get the Tor process already in whonix-gateway to be able to bootstrap via a proxy and not use the eth0 interface.
Qubes set up this proxy when it put the template into update mode (where normally the template would have no network access)
So that’s the IP address and port I need to get tor to use and bootstrap to. I would then only have one one rule in torrc that would allow the apt-get to function
[quote=“Patrick, post:92, topic:512”]I don’t understand the Qubes internals here, but I can say Tor supports all sorts of using a proxy for itself to connect.
Generally on tunneling:
Specifically Tor → proxy:
I think I have figured out a way to be able to update the templates that have no network access except for a lone http proxy. I am going to have whonix-gateway handle the proxy connection so both the gateway and workstation templates will be started with minimal firewall (only allow proxy) and no uwt (or tor).
I need to be able to disable uwt so I can run the original apt-get that will then use the http proxy that is set in /etc/apt/apt.conf.d/01qubes-proxy containing the proxy definition listed below. I don’t have the option of just running ‘/usr/bin/apt-get.anondist-orig’ but can change the link if that’s the only solution:
Should I be able to create a file ‘/etc/whonix.d/50_uwt_user’ and set
uwtwrapper_global="0"
The setting above had no effect. Changing the apt-get link to point at apt-get.anondist-orig did work, but is not preferable since if the template is shutdown prematurely, that setting will be used by the AppVMs which will break the AppVM whereas I can just link the 50_uwt_user configuration file which will only be available to the template and not the AppVM’s.
[HR]
One other question relating to Whonix build-steps.d. I created some custom build step files ‘1000_qubes-pre-config’ and ‘2900_qubes-post-config’. I noticed that there was no way to just include a ‘~/build-steps.d’ directory and place the files there although that would be a good option to have. On build, you could first move the default Whonix build-steps.d files into a ‘build-steps.d-cache’ directory (after removing anything in the cache directory), then copy files from ‘~/build-steps.d’ and even ‘/etc/build-steps.d’ into the cache directory (would allow us to over-write an existing build step that way) and then run-steps from the cache directory.
I also tried just creating a softlink to my files from the Whonix/build-steps.d directory and copying them there but they did not seem to get pickup up.
[HR]
Would you like me to create a Makefile for you? It could be used to automate many things like ‘git --init submodules’, clean, etc. I have already created a basic one that I move in place when building Whonix on Qubes, but I can make it work generically. We can add stuff like get-sources and verify sources as well so you can get rid of submodules if you wanted to and have the Makefile just grab it all and verify each package. The Qubes Makefile also lets you merge in changes, build and sign the packages is you wish.
Here is some of what I envision a Makefile would include (based on Qubes Makefile):
make gateway -- would read form a configuration file for options
make workstation -- same ---^
make get-sources -- download/update all sources (if you git rid of submodules) get+verify
make sign-all -- sign all packages
make clean-all -- remove any downloaded sources and built packages
make clean-deb -- remove any built packages
make check -- check for any uncommited changes and unsigned tags
make check-depend -- check for build dependencies ($(DEPENDENCIES))
make diff -- show diffs for any uncommitted changes
make grep RE=regexp -- grep for regexp in all components
make push -- do git push for all repos, including tags
make show-vtags -- list components version tags (only when HEAD have such) and branches
make show-authors -- list authors of Qubes code based on commit log of each component
make prepare-merge -- fetch the sources from git, but only show new commits instead of merging
make show-unmerged -- list fetched but unmerged commits (see make prepare-merge)
make do-merge -- merge fetched commits
Changing the apt-get link to point at apt-get.anondist-orig did work, but is not preferable since if the template is shutdown prematurely, that setting will be used by the AppVMs which will break the AppVM whereas I can just link the 50_uwt_user configuration file which will only be available to the template and not the AppVM's.
