sdwdate and sdwdate-gui development thread

troubadour · October 3, 2015, 8:31pm

Time sanity check on each pool after fetching times. If it fails, remove the url from valid_urls, add it to invalid_urls ang log the error.

troubadour · October 3, 2015, 8:56pm

Log slow or fast if pool timesanitycheck fails.

Patrick · October 4, 2015, 10:39am

Two items coming to my mind now before we can do an initial release.

The connection checker / wait for Tor to be ready. (Just like sdwdate-bash is doing it.)
Stream isolation settings.

If file /usr/share/anon-gw-base-files/gateway exists
→ IP → 127.0.0.1 ; Port → 9108

Otherwise (workstation)…

If folder /usr/lib/qubes-whonix exists
+IP → $(qubesdb-read /qubes-gateway) ; port → 9108
Otherwise (Non-Qubes-Whonix)…
IP → 10.152.152.10 ; port → 9108

Patrick · October 4, 2015, 11:32am

Needs to depend on python-guimessages.

Patrick · October 4, 2015, 12:10pm

With latest commits, I did run into “WARNING - Something is wrong. sdwdate could not build a list or urls.”

2015-10-04 11:51:08,667 - sdwdate.log - INFO - Last run (on Sun Oct  4 11:51:08 UTC 2015) was successful.
 
 Sleeping for 10 minutes.
2015-10-04 12:01:05,815 - sdwdate.log - INFO - Fetching remote times, start Sun Oct 04 12:01:05 UTC 2015 (unixtime 1443960065.82)
2015-10-04 12:01:05,816 - sdwdate.log - INFO - The clock is sane
 Current time Sun Oct 04 12:01:05 UTC 2015
2015-10-04 12:01:05,816 - sdwdate.log - INFO - Fetching remote times...
2015-10-04 12:01:05,816 - sdwdate.log - INFO - Running sdwdate loop, iteration 1
2015-10-04 12:01:05,816 - sdwdate.log - INFO - Requested urls ['dtsxnd3ykn32ywv6.onion', 'ymi7h25hgp3bj63v.onion', '344c6kbnjnljjzlz.onion']
2015-10-04 12:01:14,391 - sdwdate.log - INFO - Returned urls "['dtsxnd3ykn32ywv6.onion', 'ymi7h25hgp3bj63v.onion', '344c6kbnjnljjzlz.onion']"
2015-10-04 12:01:14,392 - sdwdate.log - INFO - Remote status "dtsxnd3ykn32ywv6.onion", False: connect error: 0x04: Host unreachable
2015-10-04 12:01:14,392 - sdwdate.log - INFO - Remote status "ymi7h25hgp3bj63v.onion", False: connect error: 0x04: Host unreachable
2015-10-04 12:01:14,393 - sdwdate.log - INFO - Remote status "344c6kbnjnljjzlz.onion", True
2015-10-04 12:01:14,394 - sdwdate.log - INFO - Pool 3 last url: 344c6kbnjnljjzlz.onion, web unixtime: 1443960071, web time: Sun Oct 04 12:01:11 UTC 2015, diff: -3 seconds
2015-10-04 12:01:14,394 - sdwdate.log - INFO - Running sdwdate loop, iteration 2
2015-10-04 12:01:14,394 - sdwdate.log - INFO - Requested urls ['acabtd4btrxjjrvr.onion']
2015-10-04 12:01:16,529 - sdwdate.log - INFO - Returned urls "['acabtd4btrxjjrvr.onion']"
2015-10-04 12:01:16,530 - sdwdate.log - INFO - Remote status "acabtd4btrxjjrvr.onion", True
2015-10-04 12:01:16,531 - sdwdate.log - INFO - Pool 2 last url: acabtd4btrxjjrvr.onion, web unixtime: 1443960202, web time: Sun Oct 04 12:03:22 UTC 2015, diff: 126 seconds
2015-10-04 12:01:16,532 - sdwdate.log - INFO - Running sdwdate loop, iteration 3
2015-10-04 12:01:16,532 - sdwdate.log - WARNING - Something is wrong. sdwdate could not build a list or urls.
 Restart sdwdate. If the problem persists, please report this bug.

 Sleeping for 10 minutes.

troubadour · October 4, 2015, 6:46pm

Needs to depend on python-guimessages.

Done. https://github.com/troubadoour/sdwdate/commit/8280a228eea523ff3f43dc98b64de769f5d5994d

With latest commits, I did run into "WARNING - Something is wrong. sdwdate could not build a list or urls."

This is a bug (or not) I'm looking in to.

troubadour · October 5, 2015, 8:42pm

This is a bug (or not) I’m looking in to.[/quote]
Fixed.

Patrick · October 6, 2015, 10:00am

Can you make head or tail of Whonix Forum by chance?

troubadour · October 6, 2015, 9:14pm

[quote=“Patrick, post:163, topic:1137”]- Stream isolation settings.

If file /usr/share/anon-gw-base-files/gateway exists
→ IP → 127.0.0.1 ; Port → 9108

Otherwise (workstation)…

If folder /usr/lib/qubes-whonix exists
+IP → $(qubesdb-read /qubes-gateway) ; port → 9108
Otherwise (Non-Qubes-Whonix)…
IP → 10.152.152.10 ; port → 9108[/quote]
Done. stream isolation settings · troubadoour/sdwdate@c896c2f · GitHub

- The connection checker / wait for Tor to be ready. (Just like sdwdate-bash is doing it.)

I have some trouble finding where it's done in sdwdate-bash. It's using te_pe_tb_check, I guess.

Patrick · October 6, 2015, 9:42pm

I have some trouble finding where it's done in sdwdate-bash. It's using te_pe_tb_check, I guess.

Yes.
https://github.com/Whonix/sdwdate-plugin-anon-shared-con-check/blob/55aabc3ce08aad1ceef3e48c4dbf849829bd747e/etc/sdwdate.d/31_anon_dist_con_check_plugin
+

github.com

Kicksecure/sdwdate/blob/52e09908ca9b8181de4b64a96b1abb80e4693863/usr/lib/sdwdate/modules.d/sdwdate#L1556-L1587


      
          if [ "$dispatcher_variable_name" = "DISPATCH_PREREQUISITE" ]; then
             while true; do
                sdwdate_no_duplicate_log "dispatching $dispatcher_variable_name (SDW_MODE: $SDW_MODE) (LD_PRELOAD: $LD_PRELOAD): $DISPATCH_PREREQUISITE"
          
                local prerequisite_exit_code
                prerequisite_exit_code="0"
                prerequisite_output="$(eval $dispatcher_variable_content 2>&1)" || { prerequisite_exit_code="${PIPESTATUS[0]}" ; true; };
          
                true "prerequisite_exit_code: $prerequisite_exit_code"
                true "prerequisite_output: $prerequisite_output"
          
                if [ "$prerequisite_exit_code" = "0" ]; then
                   ## Ok.
                   sdwdate_log "$dispatcher_variable_name exited $prerequisite_exit_code, continuing..."
                   break
                elif [ "$prerequisite_exit_code" = "1" ]; then
                   ## Error.
                   sdwdate_error_handler "$dispatcher_variable_name exited $prerequisite_exit_code | $prerequisite_output | requested explicit exiting..." || true
                elif [ "$prerequisite_exit_code" = "2" ]; then
                   ## Wait.

This file has been truncated. show original

Run /usr/lib/anon-shared-helper-scripts/te_pe_tb_check. Only if the file is executable. Reacting depending on the exit code of te_pe_tb_check as in above code.

Patrick · October 6, 2015, 9:48pm

Cannot use port 9108 for everyone by default. Won’t work on plain Debian. Only 9050 is available there by default. (And editing torrc is not allowed for packages.)

IP / port should be overwriteable when using it config. Currently:
PROXY_IP
PROXY_PORT
(We can consider to comment those out by default for making it easier to have the built in defaults.)

[hr]

Some more stuff to do.

random sleep as per INTERVAL, MIN_INTERVAL, RANDOMIZE
cleaning up https://github.com/troubadoour/sdwdate/blob/python/etc/sdwdate-python.d/30_sdwdate_default.conf
/etc/sdwdate-python.d → /etc/sdwdate.d [or should we avoid that for some reason?]

troubadour · October 7, 2015, 7:53pm

Run /usr/lib/anon-shared-helper-scripts/te_pe_tb_check. Only if the file is executable. Reacting depending on the exit code of te_pe_tb_check as in above code.

te_pe_tb_check returns [code]/usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 80: error: command not found[/code]

Patrick · October 7, 2015, 8:36pm

Yeah. Sorry. That script is not 100% standalone. For debugging.

+ error 'Variable TEMP_DIR is empty.' /usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 84: error: command not found

Required before running.

So you need to set that env var beforehand. The script calling te_pe_tb_check is supposed to create/delete that temp folder. However, if helpful, the script could be made completely standalone.

Btw te_pe_tb_check means,

te: Tor enabled (DisableNetwork 0 is torrc)
pe: no package manager running
tb: Tor bootstrap done.
Not the best name, but I could not find a better name that combines all these three checks.

It’s up for scrutiny.

te: Tor enabled (DisableNetwork 0 is torrc) - obviously useful, especially in case of Whonix that comes with “DisableNetwork 1” by default
pe: no package manager running - running while the package manager is running can be confusing, overlapping notifications, changing dependencies, best to wait until it’s done and start then
tb: Tor bootstrap done - Tor need to be connected before any hidden services can be accessed, so it makes sense to wait until Tor is ready before proceeding

Patrick · October 8, 2015, 12:38am

Discovered and fixed a follow up issue of Whonix Forum.

usr/lib/anon-shared-helper-scripts/te_pe_tb_check: not only check if Tor bootstrap reached 100%, but also if a Tor circuit has actually been established

https://github.com/Whonix/anon-shared-helper-scripts/commit/18807e4395642515a35a5393cd4d2d3d3cdfee2a

Please install the updated package/script since it has some potential (even small, easy change) for regressions.

[hr]

Whonix Forum made me think about something else. Wondering if the following could be viewed as a bug… sdwdate-bash does not terminate sclockadj after a sleep cycle. Should the next cycle fail, sclockadj will keep running. sclockadj is terminated after the next successful time fetching cycle and started again with the freshly obtained value. This has been reimplemented in sdwdate-python. I guess it would be better to terminate sclockadj after the sleep cycle. Because let’s say the last value was a bad one. Then the network broke down for unrelated reasons. sclockadj woul keep moving into the wrong direction then. Seems better to assign less trust to single fetched time stamps. Therefore I suggest to terminate sclockadj after the sleep cycle. What do you think?

Patrick · October 8, 2015, 11:39am

Changed that in sdwdate.

terminate sclockadj before every cycle, not just after successfully obtaining a new time stamp - https://www.whonix.org/forum/index.php/topic,1301.msg10758.html#msg10758

https://github.com/Whonix/sdwdate/commit/4c3773af11c5fed6b6a0af30e8904b53630eb8f4

And.

deactivated sclockadj_debug_helper

https://github.com/Whonix/sdwdate/commit/f8a705f3dcab93321e05e92fec0f897b28dddcf0

troubadour · October 8, 2015, 6:28pm

[quote=“Patrick, post:174, topic:1137”]Discovered and fixed a follow up issue of Whonix Forum.

Please install the updated package/script since it has some potential (even small, easy change) for regressions.[/quote]
Done.

https://www.whonix.org/forum/index.php/topic,1743.0.html made me think about something else. Wondering if the following could be viewed as a bug... sdwdate-bash does not terminate sclockadj after a sleep cycle. Should the next cycle fail, sclockadj will keep running. sclockadj is terminated after the next successful time fetching cycle and started again with the freshly obtained value. This has been reimplemented in sdwdate-python. I guess it would be better to terminate sclockadj after the sleep cycle. Because let's say the last value was a bad one. Then the network broke down for unrelated reasons. sclockadj woul keep moving into the wrong direction then. Seems better to assign less trust to single fetched time stamps. Therefore I suggest to terminate sclockadj after the sleep cycle. What do you think?

It's done that way in sdwdate-python. sclockadj is terminated after the sleep cycle, and in SIGTERM handler, in case "restart fresh" [instantly adjusting the time] is required.

Patrick · October 8, 2015, 6:50pm

You forgot to git push.

troubadour · October 8, 2015, 7:32pm

I meant sdwdate-python was already terminating sclockdj after the sleep cycle.

Regarding te_pe_tb_check, it’s called from sdwdate, tmp dir is created , “TEMP_DIR” environment is set, but it still returns

Tor is not yet fully bootstrapped. 1 % done. Tor reports: ERROR: File /tmp/tmp.GR5kjQ8dFT /tor_check_bootstrap_helper_bootstrap_file does not exist. Please report this Whonix bug! /usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 63: /tmp/tmp.GR5kjQ8dFT /tor_check_bootstrap_helper_bootstrap_file: No such file or directory /usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 63: /tmp/tmp.GR5kjQ8dFT /tor_check_bootstrap_helper_bootstrap_file: No such file or directory
It 's pushed for review: stream isolation settings · troubadoour/sdwdate@c896c2f · GitHub

Wondering if te_pe_tb_check should be rewritten in Python. We have tor_bootstrap_check.py, it’s easy to read torrc, the only unknown yet is to check package manager.

troubadour · October 8, 2015, 8:33pm

Wrong commit.
The right onet: + prerequisite check · troubadoour/sdwdate@19503b0 · GitHub

Patrick · October 8, 2015, 9:06pm

I don’t think it’s worth it to rewrite it. A lot unnecessary work. That code is very old, stable, needs little tweaking over the years. Shared with whonixcheck and tb-updater.

For debugging comment in “set -x” in /usr/lib/anon-shared-helper-scripts/te_pe_tb_check.

This is the problem.

+ rm --force '/tmp/tmp.dmqlVywOmc /tor_check_bootstrap_helper_bootstrap_file'

I.e. the environment variable ends with a new line. It should not.