upgrading from local repository for Whonix 14 wiki page

Patrick · July 11, 2017, 10:39pm

As for upgrading from local repository… I updated (/created) this page. It’s lightly tested.

https://www.whonix.org/wiki/Dev/Build_Documentation/14_deb

Probably best tested on a workstation first - potentially less troublesome.

Patrick · July 11, 2017, 10:54pm

What is not documented there yet…

One thing which makes this quite cumbersome is… You can only easily do this procedure once. Well, you can rerrun this as often as you wish, but it won’t have any effect.

It’s because when you made changes to packages and rebuild those packages, they still have the same version number. apt-get still thinks it’s the same package, hence won’t update these. Does that so far make sense?

What I did in past is artificially bump the version number. Which is scripted. Btw before attempting to script such stuff, ask me, it might be already done.

Whatever package was changed…

cd package-name
make deb-uachl-bumpup-major

To major version bump all packages at once… whonix-developer-meta-files script:

./debug-steps/packaging-helper-script pkg_need_version_bump_do

Then the above procedure of upgrading from local repository can be repeated.

However, it can be confusing when later testing to upgrade from a remote repository. These will have sane version numbers which ear earlier than the maybe many times artificially for debugging purposes only locally bumped version numbers. So you’d wipe that development VM. Cumbersome?

Perhaps it would be better during debugging to just bump the minor version number?

Anyhow. I weaseled myself out of this mess by using Qubes, where this is much more scripted.

creating / updating the local repository
trying it out in some TemplateBased VM anon-whonix-dev or sys-whonix-dev using https://github.com/Whonix/whonix-developer-meta-files/blob/master/debug-steps/qubes-repo-temp
to revert, just shut down the TemplateBased VM and go back to 1)

That of course only works for tests that don’t involve booting. But also TemplateVMs whonix-gw-dev- / whonix-ws-dev can be reverted using qvm-revert-template-changes for one change or by using git for /var/lib/qubes/vm-templates/whonix-gw-dev…

JasonJAyalaP · July 13, 2017, 2:03am

Why is the git option --jobs= whonix 14 but not 13?
Are you sure 200 jobs is a reasonable number? (froze my GW). “The number of submodules fetched at the same time.” If this is mostly network, one submodule maxes out my tor (and clearnet/github.com) bandwidth.

JasonJAyalaP · July 13, 2017, 2:08am

Something wrong with the icedove package?

Submodule path 'packages/apparmor-profile-anondist': checked out 'f5afeceb81f84b237b18973c54ff90ee5be3b8f7'
Submodule path 'packages/apparmor-profile-gwenview': checked out '5ade3b80856565cd75361b038b6a919ab7f51d49'
error: no such remote ref 6a051dc6512f8fe7daa9b5d62ec1a158ff01920b
Fetched in submodule path 'packages/apparmor-profile-icedove', but it did not contain 6a051dc6512f8fe7daa9b5d62ec1a158ff01920b. Direct fetching of that commit failed.

Did that error kill the recursive download or was it reported at the end?

I deleted the folder (in packages/) and cloned it directly fine.

JasonJAyalaP · July 13, 2017, 2:20am

after checking out 14.0.0.4.0-developers only, I did clean and submodule update. Git status shows that many packages are modified. They all seems to be the same message:

git diff packages/whonixsetup/
diff --git a/packages/whonixsetup b/packages/whonixsetup
--- a/packages/whonixsetup
+++ b/packages/whonixsetup
@@ -1 +1 @@
-Subproject commit 4a6295c7033b8c78555fc5bdf7aa758875d6d07f
+Subproject commit 4a6295c7033b8c78555fc5bdf7aa758875d6d07f-dirty

Patrick · July 13, 2017, 1:03pm

JasonJAyalaP:

Why is the git option --jobs= whonix 14 but not 13?

Whonix 13 -> Debian jessie based -> git version too old -> not
supporting --jobs.

Are you sure 200
jobs is a reasonable number?

Might be unreasonable. Always worked on my build machines which have
plenty of RAM.

Patrick · July 13, 2017, 1:09pm

JasonJAyalaP:

Something wrong with the icedove package?

Submodule path 'packages/apparmor-profile-anondist': checked out 'f5afeceb81f84b237b18973c54ff90ee5be3b8f7'
Submodule path 'packages/apparmor-profile-gwenview': checked out '5ade3b80856565cd75361b038b6a919ab7f51d49'
error: no such remote ref 6a051dc6512f8fe7daa9b5d62ec1a158ff01920b
Fetched in submodule path 'packages/apparmor-profile-icedove', but it did not contain 6a051dc6512f8fe7daa9b5d62ec1a158ff01920b. Direct fetching of that commit failed.

I added changes locally in apparmor-profile-icedove. Forgot to push.

Then added the updated git submodule to https://github.com/Whonix/Whonix
and pushed. The ref which https://github.com/Whonix/Whonix links was not
available in apparmor-profile-icedove.

Now fixed.

Did that error kill the recursive download or was it reported at the end?

It just kills the correct checkout of that very package.

I deleted the folder (in packages/) and cloned it directly fine.

In such situations please remind me to git push.

Patrick · July 13, 2017, 1:16pm

JasonJAyalaP:

after checking out 14.0.0.4.0-developers only, I did clean and submodule update. Git status shows that many packages are modified. They all seems to be the same message:
git diff packages/whonixsetup/
diff --git a/packages/whonixsetup b/packages/whonixsetup
--- a/packages/whonixsetup
+++ b/packages/whonixsetup
@@ -1 +1 @@
-Subproject commit 4a6295c7033b8c78555fc5bdf7aa758875d6d07f
+Subproject commit 4a6295c7033b8c78555fc5bdf7aa758875d6d07f-dirty

Right, such issues need to be fixed before continuation.

get into that ~/Whonix/packages/whonixsetup folder
run git status to see whats up
usually make deb-cleanup should remove all temporary files
run git status again to see if it is all sorted out

Btw whonixsetup (cli version) != whonix-setup-wizard (legacy
anon-connection-wizard and whonix setup gui version) !=
anon-connection-wizard.

Once/if anon-connection-wizard is good enough for the release of Whonix 14:

There might be legal setup that can get rid of all disclaimers, hence
whonix-setup-wizard could be abandoned.
Merge whonixsetup (cli) into anon-connection-wizard?

JasonJAyalaP · July 13, 2017, 10:38pm

In such situations please remind me to git push.

genmkfile is doing it now

It just kills the correct checkout of that very package.

Does it still work to manually clone it? What other step must be done? Remember the “attempted” checkout?

jobs

–jobs fires off more git-remote-https processes. I think this is network connections and not multiple threads handling the (i dunno, extraction?). If so, it doesn’t make sense when downloading through tor (low bandwidth) and one host (github).

git clean -ndff
git clean -dff

the first command gives an interactive prompt allowing the users to select clean or quit. Perhaps "select clean if ‘Would Remove’ should be the instructions?

run git status
usually make deb-cleanup

Now I’m seeing a bunch of

up to date with master deleted: ....

With everything (apparently) marked as deleted.

JasonJAyalaP · July 13, 2017, 10:48pm

There might be legal setup that can get rid of all disclaimers, hence
whonix-setup-wizard could be abandoned.

Are you suggesting a new tool that displays disclaimers separate from setup/anon-wizard? I agree. New ticket?

Merge whonixsetup (cli) into anon-connection-wizard?

It makes sense that the cli and gui version have the same codebase. anon-connection-wizard could detect whether or not x is running (and if qt is installed?) and load the GUI or CLI

What if running in a terminal in kde? Assume gui? I think so. Provide --cli option? Sure, assuming it’s easy to do/maintain.

You’d have 3 or 4 sections/modules in wizard:

boot (confirm root, check environment)
GUI (python qt)
CLI
Core

I don’t know how of a burden that will be on @iry. He doesn’t have to code the CLI part – nasty bash stuff, no?

Patrick · July 13, 2017, 11:35pm

JasonJAyalaP:

In such situations please remind me to git push.

genmkfile is doing it now

Fixed.

It just kills the correct checkout of that very package.

Does it still work to manually clone it?

Yes, but then you don’t have the changes that I made.

What other step must be
done? Remember the “attempted” checkout?

“Remember that something didn’t check out” suffices.

In github.com/Whonix/Whonix

git submodule update

Will sort it out.

(Perhaps also fetching latest Whonix/Whonix git master and checking that
out.)

jobs

–jobs fires off more git-remote-https processes. I think this is
network connections and not multiple threads handling the (i dunno,
extraction?).

Yes, speeds that up it.

If so, it doesn’t make sense when downloading through
tor (low bandwidth) and one host (github).

Even over Tor it’s faster than sequential.

git clean -ndff git clean -dff

the first command gives an interactive prompt allowing the users to
select clean or quit. Perhaps "select clean if ‘Would Remove’ should
be the instructions?

Seems git does that differently now.

The first command was non-interactive, just there to show what would be
removed but not remove it to prevent data loss.

Yes.

run git status usually make deb-cleanup

Now I’m seeing a bunch of

up to date with master deleted: ....

It cloned, did put the files into place, but could not checkout the
requested ref. So git now probably thinks all the files there are
extraneous files?

In that submodule… Try…

git checkout master
git reset --hard

With everything (apparently) marked as deleted.

Not sure. If above does not work, please post the full output.

Patrick · July 13, 2017, 11:42pm

JasonJAyalaP:

There might be legal setup that can get rid of all disclaimers,
hence
whonix-setup-wizard could be abandoned.

Are you suggesting a new tool that displays disclaimers separate from
setup/anon-wizard? I agree. New ticket?

Yes. That’s the plan.

tor-connection-wizard shall be non-Whonix specific, ready to be uploaded
to packages.debian.org. (Hopefully picked up by some Debian maintainer.)

Merge whonixsetup (cli) into anon-connection-wizard?

It makes sense that the cli and gui version have the same codebase.

Well, the cli version is written in bash. Probably hard to have the
code base shared with Qt for gui and cli at the same time? @ivy

anon-connection-wizard could detect whether or not x is running (and
if qt is installed?) and load the GUI or CLI

Perhaps a wrapper, yes. More of a bonus feature.

What if running in a terminal in kde? Assume gui? I think so. Provide
–cli option? Sure, assuming it’s easy to do/maintain.

No --cli option.

/usr/bin/tor-connection-wizard bash wrapper → if running in gui →
/usr/bin/tor-connection-wizard-gui

/usr/bin/tor-connection-wizard bash wrapper → if running in cli X →
/usr/bin/tor-connection-wizard-gui

/usr/bin/tor-connection-wizard bash wrapper → if running in cli virtual
terminal → /usr/bin/tor-connection-wizard-cli

Then if someone wants cli from X, they could manually start
tor-connection-wizard-cli.

You’d have 3 or 4 sections/modules in wizard: - boot (confirm root,
check environment) - GUI (python qt) - CLI - Core

I don’t know how of a burden that will be on @iry. He doesn’t have to
code the CLI part – nasty bash stuff, no?

Might be theoretic. Cli/bash part is only a stub anyhow. No no actual
bridges support. Just textual help. And probably will stay that way for
a very long time.

Could you ask please on tor-dev if adding such a wrapper / the cli stub
version would be okay being added to tor-connection-wizard or if that
would a good idea? @iry

JasonJAyalaP · July 14, 2017, 4:09am

Now it’s sdwdate-gui that’s causing an error. We can I instructions about this error, unless you think there can be an automatic/foolproof way to prevent it.

Everything in multiple (but not all) packages are … weird…

git status shows “up to date with master”, changes to be commited “deleted …” everything, in green. Simply typing “git checkout master” (without reseting) says “already on master” and git status shows “working tree clean” … Weird.

I did “git checkout master” in every packages/ subfolder

find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && pwd" \;

then re-ran

git submodule update --init --recursive

That got me to a clean detached head at 14.0.0.4.0-developers

So, the folders that don’t submodule update stay at master… but somehow complain to Whonix/Whonix ?

JasonJAyalaP · July 14, 2017, 4:26am

I’m running into an error with $apt_get_update_wrapper_source_path_full “${APTGETOPT[@]}” update

############################################################
ERROR in ./build-steps.d/1100_prepare-build-machine detected!
anon_dist_build_version: 
(whonix_build_error_counter: 1)
(benchmark: 00:00:29)
trap_signal_type_previous: unset
trap_signal_type_last    : ERR
process_backtrace_result:
1: : /sbin/init 
2: : kdeinit5: Running...                                   
3: : /usr/bin/ksmserver 
4: : /usr/bin/plasmashell --shut-up 
5: : /usr/bin/konsole 
6: : /bin/bash 
7: : sudo -E ./build-steps.d/1100_prepare-build-machine --internalrun --build --target root 
8: : /bin/bash ./build-steps.d/1100_prepare-build-machine --internalrun --build --target root 
function_trace_result:
main (line number: 376)
main (line number: 371)
build_machine_setup (line number: 265)
errorhandlergeneral (line number: 335)
errorhandlerprocessshared (line number: 170)
last_failed_bash_command: $apt_get_update_wrapper_source_path_full "${APTGETOPT[@]}" update
last_failed_exit_code: 125
ERROR in ./build-steps.d/1100_prepare-build-machine detected!
############################################################

Patrick · July 15, 2017, 12:52pm

Full output required. Cannot make head or tail of the description.

Patrick · July 21, 2017, 12:03pm

cd /tmp
/tmp $ mkdir xx
/tmp $ cd xx
/tmp/xx $ git clone --jobs=200 --recursive https://github.com/Whonix/Whonix

cd Whonix
/tmp/xx/Whonix $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)

...
        modified:   packages/whonix-welcome-page (modified content)
...

no changes added to commit (use "git add" and/or "git commit -a")
  
  /tmp/xx/Whonix $ cd packages/whonix-welcome-page/
  /tmp/xx/Whonix/packages/whonix-welcome-page $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        deleted:    CONTRIBUTING.md
        deleted:    COPYING
        deleted:    GPLv3
        deleted:    Makefile
        deleted:    README.md
        deleted:    changelog.upstream
        deleted:    debian/changelog
        deleted:    debian/compat
        deleted:    debian/control
        deleted:    debian/copyright
        deleted:    debian/rules
        deleted:    debian/source/format
        deleted:    debian/source/lintian-overrides
        deleted:    debian/watch
        deleted:    etc/X11/Xsession.d/20whonix-welcome-page
        deleted:    etc/profile.d/20_whonix-welcome-page.sh
        deleted:    usr/lib/whonix-welcome-page/env_var.sh
        deleted:    usr/share/homepage/whonix-welcome-page/logo.png
        deleted:    usr/share/homepage/whonix-welcome-page/stylesheet.css
        deleted:    usr/share/homepage/whonix-welcome-page/whonix.html

  /tmp/xx/Whonix/packages/whonix-welcome-page $ git reset --hard
HEAD is now at 1c73d56 bumped changelog version
  /tmp/xx/Whonix/packages/whonix-welcome-page $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
  /tmp/xx/Whonix/packages/whonix-welcome-page $ 
  /tmp/xx/Whonix/packages/whonix-welcome-page $ cd ..
  /tmp/xx/Whonix/packages $ git status

whonix-welcome-page fixed.

Patrick · July 21, 2017, 12:09pm

During git clone…

Fetched in submodule path 'packages/whonix-gw-firewall', but it did not contain dc7e553f637ebfc4e0735438b830fea22663075b. Direct fetching of that commit failed.

This was the cause.

https://github.com/Whonix/whonix-gw-firewall/commit/99a0e1eb3ce3bd1728f12f673522d239b0a3d9f6

You changed that repository, but then didn’t add the updated git commit to https://github.com/Whonix/Whonix. Therefore from git’s perspective of https://github.com/Whonix/Whonix, whonix-gw-firewall was ahead.

On the other hand from my local (!) Whonix/Whonix perspective, everything was up to date.

Patrick · July 21, 2017, 12:13pm

To fix the many

    modified:   packages/.... (modified content)

messages, I did…

In https://github.com/Whonix/Whonix/blob/master/help-steps/cleanup-files#L41 under git clean -d --force --force -x, I’ve added git reset --hard. Then did run from Whonix/Whonix, ./help-steps/cleanup-files.

But anyhow. Fresh git clone --jobs=200 --recursive https://github.com/Whonix/Whonix is now fixed as well.

In future, we need to better coordinate changes. Whenever a package was changed, I need to be somehow informed. Like with https://github.com/Whonix/whonix-gw-firewall/commit/99a0e1eb3ce3bd1728f12f673522d239b0a3d9f6. So I can git merge locally as well. And so we don’t forget to update Whonix/Whonix.

JasonJAyalaP · September 4, 2017, 10:08pm

I’m almost certain that jobs=200 is an unreasonable number. All the git-http processes hard freeze the OVAs. I tried to find examples on google but it seems unsearchable. I did find someone doing jobs=4. If there’s no objection, I will change it to that.

Patrick · September 5, 2017, 10:03am

JasonJAyalaP:

I’m almost certain that jobs=200 is an unreasonable number. All the git-http processes hard freeze the OVAs. I tried to find examples on google but it seems unsearchable. I did find someone doing jobs=4. If there’s no objection, I will change it to that.

If that’s the lowest number that doesn’t freeze… Yes, change it.