Let me rephrase that.
Running the script with the log option is essential for troubleshooting.
Occam’s razor.
It’s not in any way personal, it’s just the most practical way of dealing with things. It’s however not perfect as we saw.
Done.
I deleted it already and to make sure there's nothing in there, I'm going to read it whole through, manually. If there is something identifying in there, I'm getting back to you, if not, screw it.This review is also a great contribution.
(2) I dislike not being taken seriously. I like Whonix and I would even contribute much more (if appreciated) but if I tell you that I haven't made a pebkac mistake (like forgetting to install "locales", checking out the wrong branches, typos in my modifications, etc.), I dislike being asked for another and another time. There is a reason I say that the build script is failing and it's not me as I have very carefully examined (in this very specific case) if it's my mistake before even posting that it's failing. (to 2) being treated like a user/newbie, call it whatever you like - instead of a peer - contributed very much to a 2-day build-horror that could have been avoided if my very first mention of locale failing would have been taken seriously, looked at, thought about why it's producing errors. At the end of the day, the build instructions were plain wrong (UK) and nothing on my part and building this beast three times in a row on bare metal hardware with 3 re-installs is pita (unnecessary pita).I am sorry about this. I am not a professional developer. I am a self-taught hobbyists and working on improving the build process. This is the first Free Software projects I am maintaining (my smaller ones such as VPN-Firewall don't really count). Due to time constraints (and perhaps also brain limitations) I won't be able to provide instructions/code that is near perfect for a testers-only release. And I am only steps away from not being able to produce near perfect product once stable. I mean, the Whonix 7 build instructions did break over time. A lot effort has been invested to make the build script more robust. Perhaps one day that goal can be reached.
I commented on the uuids. I know what a uuid is (again, I'm not stupid)I don't know, that you know what they are about. And I don't expect anyone to know.
[quote]This isn't a game, where small differences in the gameplay don't matter. (...) Advanced Linux users as well as testers require a high frustration tolerance. Welcome to Free Software development. And you can always claim your money back, if you're not satisfied.[/quote] How to comment here? I put it that way: would you contribute to a software project were you need to build something four times in a row, be treated like stupid: "Welcome to Free Software Development"? < trust me, I'm way longer in the free software world than you (presumably) are and do not like to be treated arrogant. OK? Fact is, a volunteer tester loses 2 full working days to test your build, get questions asked like "are you sure you're not stupid and can read English?", i.e. have you installed "locales" - you asked me several times this question, last time via a command I understand (no worries), typos, faulty modifications ... at the end of the day it's nothing of that but your instructions and/or your script. If you want to work like that, fine, I can contribute to other projects then. If you'd like to start to talk to me respectfully, I'm gladly in.Most likely (at least part of the story) we get into conflict here as writing is such a bad medium for something like that. I can tell you that I do not feel treated the way I want here. Re-read what you said, especially the last quote and several other things along those lines - the tone is important here (I know my tone isn’t cool as well at the moment). Anyways, I hope we get in sync again as I’m not at all interested in this to be honest but in productive contribution (I’m sure you’re on the same page here).
To sum up: I make mistakes as everybody does but if I assure you that I did something correct, please do not ask me another and another time the same thing and let me re-install an OS 3 times. You know, I very much know what “sudo which update-locale ; echo $?” actually means. The very same as “do you have “locales” installed”? - are you kidding me? It’s a nice try, but you need to try harder here or better stop it and treat contributors like peers, not idiots.
This is the way I diagnose bugs. I had absolutely no idea what could have caused it. That script worked for a long time. I can imagine, you really installed the grml_packages list, you really have locales installed and for some strange reason, “update-locale” is still not available. It’s unlikely (ex: hdd corruption), but not impossible. And when I tried it myself, I made a mistake and never noticed “United States” in instructions has changed to “United Kingdom”. So I tested with “United States” and never experienced that bug myself. After having excluded everything, my brain finally came up with an idea, which was the answer to this mystery.
If you take offense by “sudo which update-locale ; echo $?”, I can’t promise to do better. This is the way I diagnose bugs. I can promise, that it’s not my intention to offend anyone. For me, there is no point in offense. I’ll do this for the sake of coding for fun, for a good cause, and side effects. You’re welcome to contribute further. See my failure to communicate as deficiency in coding/social skills.
@Occq
I’m on the same page that logs are essential for troubleshooting. I refer to #40 and after finally understanding the actual error, even much earlier. But, regarding the actual build error, everything is cool. You made a mistake. We all do. Period. I appreciate the way you dealt with it, i.e. briefly apologizing and that’s it.
@adrelanos
Thanks for explaining thoroughly the whonix repositories. I’m going to re-subscribe to testing and switch to stable again when Whonix 8 will be released. As I understand it, it very much works like Debian stable, testing, sid from a release management perspective. As of you mentioning that Whonix and Debian repos are distinct things … see my writeup above.
@all
I have another question + another remark regarding the build process:
- I noticed that it creates a swap file at /swapfile1. Why is this?
- I mentioned earlier that the build script messed up my network configuration slightly. That is to say, it deleted my wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf line for the WAN interface (not a big issue). I’m not sure how the script replaces the /etc/network/interfaces.whonix, most likely it completely re-writes it. I’m just saying that we may investigate how to preserve specifics in particular network configurations while building without actually erasing them, i.e. by carefully parsing them, merging them - haven’t thought about this for too long just yet. Just thinking it would be a potential improvement to the build process. Might be confusing to newbies otherwise.
For VM builds, no swapfile is automatically created. Do you propose shipping by default without a swapfile?
It would easy to disable swapfile creation for physical isolation users.
(And for modders sake: there is a build configuration to disable chroot / postinst.d scripts.)
This is very strange. I have no idea how this can happen. There is no result of “grep -r supplicant *” in Whonix’s code. Perhaps by replacing /etc/network/interfaces and first boot, something automatically deletes /etc/wpa_supplicant/wpa_supplicant.conf. But this also shouldn’t happen, because by Debian policy, /etc is only for users and packages are not allowed to edit or even delete them (with the exception for initial creation).
Using config-package-dev (from jessie: https://packages.debian.org/jessie/config-package-dev). See also debian/*.displace files. config-package-dev helps to take ownership of files, which are owned by other packages, such as /etc/tor/torrc is owned by the “tor” Debian package. When upstream (Debian) releases upgraded, /etc/tor/torrc.whonix-orig gets upgraded, /etc/tor/torrc stays untouched. Only Whonix’s package can update it. config-package-dev implementes this using symlinks and Debian maintainer scripts.
The story behind this:
Whonix’s focus is on virtual machine images. Physical Isolation is something that can be maintained as well with reasonable effort. I am not good at parsing/grepping/merging/grep/awk/sed/replace. To keep things simple and maintainable, config-package-dev seemed like the best solution. So upstream (Debian) can’t mess up these files. And when there is need for a security update, new features or just better comments, it’s simple to ship a new version of these files. I see for physical isolation users it’s a more difficult, but whole physical isolation is more difficult.
If you have any suggestions, I am eager to hear them.
For VM builds, no swapfile is automatically created. Do you propose shipping by default without a swapfile? It would easy to disable swapfile creation for physical isolation users.Your decision after all. I would say disabling it would be good (although it doesn't harm the machine). When building for physical isolation, the host/target is prepared including swapfile and/or -partition prior building. Thus I see no need to auto-create it. Not a big deal though - just reporting.
(And for modders sake: there is a build configuration to disable chroot / postinst.d scripts.)(I think) it's also mentioned somewhere in the build instructions. Could be useful for further builds. Thanks for the hint.
This is very strange. I have no idea how this can happen. There is no result of "grep -r supplicant *" in Whonix's code. Perhaps by replacing /etc/network/interfaces and first boot, something automatically deletes /etc/wpa_supplicant/wpa_supplicant.conf. But this also shouldn't happen, because by Debian policy, /etc is only for users and packages are not allowed to edit or even delete them (with the exception for initial creation).We (just slightly) misunderstood each other here. It's not deleting the actual /etc/wpa_supplicant/wpa_supplicant.conf file but the line "wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf" within the actual /etc/network/interfaces, which is a symlink to /etc/network/interfaces.whonix after building. My config (WAN part) prior building read: [code]auto wlan0 iface wlan0 inet dhcp wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf[/code] afterwards it read: [code]auto wlan0 iface wlan0 inet dhcp pre-up /usr/bin/whonix_firewall[/code] with the wpa-conf deleted/omitted. Also not a big deal. I just consider it thorough bug reporting to mention it. Fixed it by reintroducing the wpa-conf and running [code]sudo service networking restart[/code]
Using config-package-dev (from jessie: https://packages.debian.org/jessie/config-package-dev). See also debian/*.displace files. config-package-dev helps to take ownership of files, which are owned by other packages, such as /etc/tor/torrc is owned by the "tor" Debian package. When upstream (Debian) releases upgraded, /etc/tor/torrc.whonix-orig gets upgraded, /etc/tor/torrc stays untouched. Only Whonix's package can update it. config-package-dev implementes this using symlinks and Debian maintainer scripts.I understand. Basically "dpkg-diverting" configuration files shipped by conflicting packages. Thanks for explaining.
Whonix's focus is on virtual machine images. Physical Isolation is something that can be maintained as well with reasonable effort. I am not good at parsing/grepping/merging/grep/awk/sed/replace. To keep things simple and maintainable, config-package-dev seemed like the best solution. So upstream (Debian) can't mess up these files. And when there is need for a security update, new features or just better comments, it's simple to ship a new version of these files. I see for physical isolation users it's a more difficult, but whole physical isolation is more difficult.I'm perfectly on the same page here. Thanks actually for implementing it that deliberated. dpkg-divert is the most clean way to deal with it (at least what I can currently think of).
As of suggestions: we may, at some point, introduce some awk/sed parsing of the existing (prior building) network configuration to avoid omitting something important like wpa-conf, i.e. noticing it’s there and re-introducing it to /etc/network/interfaces_whonix. I’m going to think about that one. I mean, I actually assumed it would happen before building and it did exactly that. Newbies could be confused by that - thus I reported it to you. Although: newbies do not build physical isolation anyways 
Quoting myself:
Your decision after all. I would say disabling it would be good (although it doesn't harm the machine). When building for physical isolation, the host/target is prepared including swapfile and/or -partition prior building. Thus I see no need to auto-create it. Not a big deal though - just reporting.Or better: What is your actual reasoning to introduce a swapfile when buliding for physical isolation? Maybe I miss something here.
[quote=“Cerberus, post:85, topic:67”][quote]For VM builds, no swapfile is automatically created. Do you propose shipping by default without a swapfile?
It would easy to disable swapfile creation for physical isolation users.[/quote]
Your decision after all. I would say disabling it would be good (although it doesn’t harm the machine). When building for physical isolation, the host/target is prepared including swapfile and/or -partition prior building. Thus I see no need to auto-create it. Not a big deal though - just reporting.[/quote]
Sure. Will disable it for Whonix 8 + 1 for physical isolation users. Do you wish to get credit in git commit logs?
We (just slightly) misunderstood each other here. It's not deleting the actual /etc/wpa_supplicant/wpa_supplicant.conf fileThanks, $DEITY!
but the line "wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf" within the actual /etc/network/interfaces, which is a symlink to /etc/network/interfaces.whonix after building.whonix_(gateway|workstation)/etc/network/interfaces.whonix becomes your /etc/network/interfaces after building.
Your /etc/network/interfaces before building becomes your /etc/network/interfaces.whonix-orig after building.
That’s why your wpa-conf line is lost, I think.
This inconvenience derives from the way how VM images are build. Having a “don’t replace /etc/network/interfaces when installing whonix-gateway-files” when variable BARE_METAL=1" in the code would be nice, but I wouldn’t know how to implement it. For Whonix 8 + 1 or even later, I plan to split Whonix’s source code into smaller source and Debian packages. Then whonix_(gateway|workstation)/etc/network/interfaces.whonix config files, could be put into a whonix-(gateway|workstation)-network-interfaces package, which is omitted for physical isolation users.
Also not a big deal. I just consider it thorough bug reporting to mention it. Fixed it by reintroducing the wpa-conf and runningI can imagine that this is confusing. It must also look quite dilettantish, that your network is non-functional after installing. What we could add to physical isolation instructions in meanwhile "cp /etc/network/interfaces.whonix-orig /etc/network/interfaces" to get your old configuration back. The other option, editing /etc/network/interfaces.whonix before building and committing it to git seems even more awkward.
Basically "dpkg-diverting" configuration files shipped by conflicting packages.Yes. It automates dpkg-divert this and their package seems quite stable. Also they have a good support mailing list.
As of suggestions: we may, at some point, introduce some awk/sed parsing of the existing (prior building) network configuration to avoid omitting something important like wpa-conf, i.e. noticing it's there and re-introducing it to /etc/network/interfaces_whonix. I'm going to think about that one. I mean, I actually assumed it would happen before building and it did exactly that. Newbies could be confused by that - thus I reported it to you. Although: newbies do not build physical isolation anyways :PWhat do you think about my method of splitting Whonix into several packages?
Sure. Will disable it for Whonix 8 + 1 for physical isolation users. Do you wish to get credit in git commit logs?Great, thanks. Getting credits isn't required here. I'm just reporting very minor things as a part of testing. Appreciate the offer though.
I can imagine that this is confusing. It must also look quite dilettantish, that your network is non-functional after installing. What we could add to physical isolation instructions in meanwhile "cp /etc/network/interfaces.whonix-orig /etc/network/interfaces" to get your old configuration back. The other option, editing /etc/network/interfaces.whonix before building and committing it to git seems even more awkward.Actually I very much appreciate your efforts to automate this (as I appreciate Whonix overall). Not amateurish at all from my perspective. Escaping special configurations would certainly be the very best thing (but needs to be scripted. If I'm completely setup, I'm going to look at it again. Maybe I can contribute something here) with the current approach quite close to perfect. Disabling the automatic replacement of /etc/network/interfaces wouldn't be superior here. I'm completely on the same page here. Let's leave it as is - just mentioning it. Maybe you haven't noticed before. If someone, like me, needs to introduce "scary", i.e. non-std. things like "wpa-conf" he/she needs to bite the bullet - anyways a very minor issue that's easily fixed.
What do you think about my method of splitting Whonix into several packages?Like that! To make clear: I was not about criticizing Whonix, just the testing process we have passed during the weekend. While my opinion isn't very much relevant here, you're doing a _great_ job with Whonix from my perspective. --- Now, I'm sorry to bother again ;) I'm currently trying to install the Workstation host through the new gateway and I'm having issues here. This is most likely a silly mistake I make, but here it goes: I configured the workstation host, i.e. the Debian install with
IP/netmask: 192.168.0.11/24 Gateway/DNS: 192.168.0.10and I'm trying to enable Debian repos during installation. Problem is: it seems to lack connectivity through the gateway, i.e. the host/debian install isn't able to reach the net through the gateway. May you please briefly help me troubleshooting here? Is the IP/netmask, Gateway/DNS configuration of the host that's installing as it's supposed to be? I checked for typos here, nothing found and gateway is at 192.168.0.10/24. Debian install is complaining about not being able to connect to the net, while gateway tells me "timesync fine, connected to Tor". I tried to ping in both directions > no connectivity. Now, it's either me failing (stupid mistake), a faulty cable or something else. Any feedback? Thanks!
I don’t know if you’re following out other thread as well. Just FYI, much better error handling has been added:
The old script behavior was unbearable indeed.
To make clear: I was not about criticizing Whonix,Wasn't perceived as criticism. Not in any negative sense.
I tried to ping in both directions > no connectivity.About ping, I just updated this FAQ entry a bit: https://www.whonix.org/wiki/FAQ#Why_can.27t_I_ping_the_Whonix-Gateway.3F
Ping workstation host → gateway won’t work as long as the firewall is active. The other way around gateway ping → workstation however should work (quick test in a VM, though).
I configured the workstation host, i.e. the Debian install with ... and I'm trying to enable Debian repos during installationI don't know. I never made friends with network settings during installation. There could even be bugs in the installer. What you could try is finishing without Debian repos during installation and see if you can manually get networking working at all. To exclude the possibility, that something is wrong with the gateway. (Or for a quick test boot a Live CD for that purpose and see if the gateway will proxy traffic for you.)
address 192.168.0.11
netmask 255.255.255.0
gateway 192.168.0.10
There are other issues with the build environment.
[ul][li]Building in LXDE fails. Probably some device auto mounter gets in the way.[/li]
[li]Likewise building in GNOME fails. Probably some device auto mounter gets in the way. We had a workaround in https://www.whonix.org/wiki/Dev/BuildDocumentation_8 using “gsettings set org.gnome.desktop.media-handling automount-open false” but it doesn’t seem to work in wheezy (worked in jessie).[/li]
[li]When building VM images when one hasn’t chosen en-US while installing, the following warning will be shown by apt-get:[/li][/ul]
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "",
LC_ALL = (unset),
LANG = "en_GB.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
And this also leads to debsums output in /usr/share/whonix/chroot-scripts-pre.d/20_sanity_checks where none is expected, breaking the build.
Found a solution for the latter (https://github.com/Whonix/Whonix/commit/d6cf74e843e98f0b6760a4c678a6d1f486993c92).
@adrelanos
lol, you know what is going on here with the network connection? It’s Murphy’s law 
that is to say, the gateway + the workstation host both happen to have network cards that do not support Auto-MDIX (automatic medium-dependent interface crossover). I need a crossover cable to connect them. bummer! Took me hours to figure that out.
connected gateway to workstation host with 3 distinct cables: link not ready
connected test-laptop to gateway: link up
connected test-laptop to workstation host: link up
interesting indeed 
then the proverbial lightbulb appeared above my head! At least that is my conclusion so far. now, I’m going to research if I can control this with the kernel module somehow, otherwise use a switch, or insert another network card supporting Auto-MDIX into one of the two rigs.
I thought I share it with you to provide a good laugh to you!
rofl … I promise: if you considered my previous post funny, now comes comedy! So, without any further ado … Here it comes (I’m telling the story in a third person perspective as I consider it funnier that way):
OK, so, the situation is: Cerberus recognized (after hours) that he/she either needs a crossover cable or a switch, swap the card within one of the rigs or find a way to configure the kernel module(s). Status quo, see above.
Ok, fine. 1st decision? We’re pros right? -> so, kernel module. What a bad idea! Some time later into the process: %RfgHN/$Ws network cards. LOL.
2nd decision? A switch is lame right? -> so, swapping the card in the workstation host. Again, bad idea!
1st try: Cerberus finds an old network card that he/she knows supports Auto-MDIX (let me mention here that one card is usually enough here to auto-sense). Now, Cerberus disassembles the workstation host, swaps cards, put it all together again, switch on -> hardware dead! LOL.
2nd try: Searching for cards is on the agenda, again. Cerberus finds another card, searches the web for Auto-MDIX support -> positive! Nice! disassembling the workstation host for a second time, swapping cards, switching on -> link detected. Yeah! Problem solved? Unfortunately NO as the gateway side couldn’t care less -> no link. WTF? I still don’t know how this can happen but obviously the DFS§$wghr5 cards may work with a card attached that supports Auto-MDIX (see test-laptop) but not with another one (that also is supposed to support it and shows “link up”). Great!
Now? 3rd decision? Switch (btw, the most easiest - Cerberus does not have a crossover cable - but least appreciated solution here). Worked on first try! Could have been easy.
Hope you have a good laugh!
I don’t find such issues funny. I am not sure, I would have come to the idea, what the actual cause is.
Thanks for reporting. I created a troubleshooting chapter on the physical isolation page.
Those hints may help someone save some time since you shared these possibly sources of errors with us.