[HOME] [DOWNLOAD] [DOCS] [NEWS] [SUPPORT] [TIPS] [ISSUES] [Priority Support]

sdwdate and sdwdate-gui development thread


#161

Time sanity check on each pool after fetching times. If it fails, remove the url from valid_urls, add it to invalid_urls ang log the error.


#162

Log slow or fast if pool timesanitycheck fails.


#163

Two items coming to my mind now before we can do an initial release.

  • The connection checker / wait for Tor to be ready. (Just like sdwdate-bash is doing it.)

  • Stream isolation settings.

If file /usr/share/anon-gw-base-files/gateway exists
-> IP -> 127.0.0.1 ; Port -> 9108

Otherwise (workstation)…

If folder /usr/lib/qubes-whonix exists
+IP -> $(qubesdb-read /qubes-gateway) ; port -> 9108
Otherwise (Non-Qubes-Whonix)…
IP -> 10.152.152.10 ; port -> 9108


#164

Needs to depend on python-guimessages.


#165

With latest commits, I did run into “WARNING - Something is wrong. sdwdate could not build a list or urls.”

2015-10-04 11:51:08,667 - sdwdate.log - INFO - Last run (on Sun Oct  4 11:51:08 UTC 2015) was successful.
 
 Sleeping for 10 minutes.
2015-10-04 12:01:05,815 - sdwdate.log - INFO - Fetching remote times, start Sun Oct 04 12:01:05 UTC 2015 (unixtime 1443960065.82)
2015-10-04 12:01:05,816 - sdwdate.log - INFO - The clock is sane
 Current time Sun Oct 04 12:01:05 UTC 2015
2015-10-04 12:01:05,816 - sdwdate.log - INFO - Fetching remote times...
2015-10-04 12:01:05,816 - sdwdate.log - INFO - Running sdwdate loop, iteration 1
2015-10-04 12:01:05,816 - sdwdate.log - INFO - Requested urls ['dtsxnd3ykn32ywv6.onion', 'ymi7h25hgp3bj63v.onion', '344c6kbnjnljjzlz.onion']
2015-10-04 12:01:14,391 - sdwdate.log - INFO - Returned urls "['dtsxnd3ykn32ywv6.onion', 'ymi7h25hgp3bj63v.onion', '344c6kbnjnljjzlz.onion']"
2015-10-04 12:01:14,392 - sdwdate.log - INFO - Remote status "dtsxnd3ykn32ywv6.onion", False: connect error: 0x04: Host unreachable
2015-10-04 12:01:14,392 - sdwdate.log - INFO - Remote status "ymi7h25hgp3bj63v.onion", False: connect error: 0x04: Host unreachable
2015-10-04 12:01:14,393 - sdwdate.log - INFO - Remote status "344c6kbnjnljjzlz.onion", True
2015-10-04 12:01:14,394 - sdwdate.log - INFO - Pool 3 last url: 344c6kbnjnljjzlz.onion, web unixtime: 1443960071, web time: Sun Oct 04 12:01:11 UTC 2015, diff: -3 seconds
2015-10-04 12:01:14,394 - sdwdate.log - INFO - Running sdwdate loop, iteration 2
2015-10-04 12:01:14,394 - sdwdate.log - INFO - Requested urls ['acabtd4btrxjjrvr.onion']
2015-10-04 12:01:16,529 - sdwdate.log - INFO - Returned urls "['acabtd4btrxjjrvr.onion']"
2015-10-04 12:01:16,530 - sdwdate.log - INFO - Remote status "acabtd4btrxjjrvr.onion", True
2015-10-04 12:01:16,531 - sdwdate.log - INFO - Pool 2 last url: acabtd4btrxjjrvr.onion, web unixtime: 1443960202, web time: Sun Oct 04 12:03:22 UTC 2015, diff: 126 seconds
2015-10-04 12:01:16,532 - sdwdate.log - INFO - Running sdwdate loop, iteration 3
2015-10-04 12:01:16,532 - sdwdate.log - WARNING - Something is wrong. sdwdate could not build a list or urls.
 Restart sdwdate. If the problem persists, please report this bug.

 Sleeping for 10 minutes.

#166
Needs to depend on python-guimessages.
Done. https://github.com/troubadoour/sdwdate/commit/8280a228eea523ff3f43dc98b64de769f5d5994d
With latest commits, I did run into "WARNING - Something is wrong. sdwdate could not build a list or urls."
This is a bug (or not) I'm looking in to.

#167

This is a bug (or not) I’m looking in to.[/quote]
Fixed.


#168

Can you make head or tail of https://www.whonix.org/forum/index.php/topic,1743.0.html by chance?


#169

[quote=“Patrick, post:163, topic:1137”]- Stream isolation settings.

If file /usr/share/anon-gw-base-files/gateway exists
-> IP -> 127.0.0.1 ; Port -> 9108

Otherwise (workstation)…

If folder /usr/lib/qubes-whonix exists
+IP -> $(qubesdb-read /qubes-gateway) ; port -> 9108
Otherwise (Non-Qubes-Whonix)…
IP -> 10.152.152.10 ; port -> 9108[/quote]
Done. https://github.com/troubadoour/sdwdate/commit/c896c2f527844a70e43f5c0b130c65a326a63141

- The connection checker / wait for Tor to be ready. (Just like sdwdate-bash is doing it.)
I have some trouble finding where it's done in sdwdate-bash. It's using te_pe_tb_check, I guess.

#170
I have some trouble finding where it's done in sdwdate-bash. It's using te_pe_tb_check, I guess.

Yes.


+

Run /usr/lib/anon-shared-helper-scripts/te_pe_tb_check. Only if the file is executable. Reacting depending on the exit code of te_pe_tb_check as in above code.


#171

Cannot use port 9108 for everyone by default. Won’t work on plain Debian. Only 9050 is available there by default. (And editing torrc is not allowed for packages.)

IP / port should be overwriteable when using it config. Currently:
PROXY_IP
PROXY_PORT
(We can consider to comment those out by default for making it easier to have the built in defaults.)

[hr]

Some more stuff to do.


#172
Run /usr/lib/anon-shared-helper-scripts/te_pe_tb_check. Only if the file is executable. Reacting depending on the exit code of te_pe_tb_check as in above code.
te_pe_tb_check returns [code]/usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 80: error: command not found[/code]

#173

Yeah. Sorry. That script is not 100% standalone. For debugging.

+ error 'Variable TEMP_DIR is empty.' /usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 84: error: command not found

Required before running.

So you need to set that env var beforehand. The script calling te_pe_tb_check is supposed to create/delete that temp folder. However, if helpful, the script could be made completely standalone.

Btw te_pe_tb_check means,

  • te: Tor enabled (DisableNetwork 0 is torrc)
  • pe: no package manager running
  • tb: Tor bootstrap done.
    Not the best name, but I could not find a better name that combines all these three checks.

It’s up for scrutiny.

  • te: Tor enabled (DisableNetwork 0 is torrc) - obviously useful, especially in case of Whonix that comes with “DisableNetwork 1” by default
  • pe: no package manager running - running while the package manager is running can be confusing, overlapping notifications, changing dependencies, best to wait until it’s done and start then
  • tb: Tor bootstrap done - Tor need to be connected before any hidden services can be accessed, so it makes sense to wait until Tor is ready before proceeding

#174

Discovered and fixed a follow up issue of https://www.whonix.org/forum/index.php/topic,1743.0.html.

usr/lib/anon-shared-helper-scripts/te_pe_tb_check: not only check if Tor bootstrap reached 100%, but also if a Tor circuit has actually been established
https://github.com/Whonix/anon-shared-helper-scripts/commit/18807e4395642515a35a5393cd4d2d3d3cdfee2a

Please install the updated package/script since it has some potential (even small, easy change) for regressions.

[hr]

https://www.whonix.org/forum/index.php/topic,1743.0.html made me think about something else. Wondering if the following could be viewed as a bug… sdwdate-bash does not terminate sclockadj after a sleep cycle. Should the next cycle fail, sclockadj will keep running. sclockadj is terminated after the next successful time fetching cycle and started again with the freshly obtained value. This has been reimplemented in sdwdate-python. I guess it would be better to terminate sclockadj after the sleep cycle. Because let’s say the last value was a bad one. Then the network broke down for unrelated reasons. sclockadj woul keep moving into the wrong direction then. Seems better to assign less trust to single fetched time stamps. Therefore I suggest to terminate sclockadj after the sleep cycle. What do you think?


#175

Changed that in sdwdate.

terminate sclockadj before every cycle, not just after successfully obtaining a new time stamp - https://www.whonix.org/forum/index.php/topic,1301.msg10758.html#msg10758
https://github.com/Whonix/sdwdate/commit/4c3773af11c5fed6b6a0af30e8904b53630eb8f4

And.

deactivated sclockadj_debug_helper
https://github.com/Whonix/sdwdate/commit/f8a705f3dcab93321e05e92fec0f897b28dddcf0

#176

[quote=“Patrick, post:174, topic:1137”]Discovered and fixed a follow up issue of https://www.whonix.org/forum/index.php/topic,1743.0.html.

Please install the updated package/script since it has some potential (even small, easy change) for regressions.[/quote]
Done.

https://www.whonix.org/forum/index.php/topic,1743.0.html made me think about something else. Wondering if the following could be viewed as a bug... sdwdate-bash does not terminate sclockadj after a sleep cycle. Should the next cycle fail, sclockadj will keep running. sclockadj is terminated after the next successful time fetching cycle and started again with the freshly obtained value. This has been reimplemented in sdwdate-python. I guess it would be better to terminate sclockadj after the sleep cycle. Because let's say the last value was a bad one. Then the network broke down for unrelated reasons. sclockadj woul keep moving into the wrong direction then. Seems better to assign less trust to single fetched time stamps. Therefore I suggest to terminate sclockadj after the sleep cycle. What do you think?
It's done that way in sdwdate-python. sclockadj is terminated after the sleep cycle, and in SIGTERM handler, in case "restart fresh" [instantly adjusting the time] is required.

#177

You forgot to git push.


#178

I meant sdwdate-python was already terminating sclockdj after the sleep cycle.

Regarding te_pe_tb_check, it’s called from sdwdate, tmp dir is created , “TEMP_DIR” environment is set, but it still returns

Tor is not yet fully bootstrapped. 1 % done. Tor reports: ERROR: File /tmp/tmp.GR5kjQ8dFT /tor_check_bootstrap_helper_bootstrap_file does not exist. Please report this Whonix bug! /usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 63: /tmp/tmp.GR5kjQ8dFT /tor_check_bootstrap_helper_bootstrap_file: No such file or directory /usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 63: /tmp/tmp.GR5kjQ8dFT /tor_check_bootstrap_helper_bootstrap_file: No such file or directory
It 's pushed for review: https://github.com/troubadoour/sdwdate/commit/c896c2f527844a70e43f5c0b130c65a326a63141

Wondering if te_pe_tb_check should be rewritten in Python. We have tor_bootstrap_check.py, it’s easy to read torrc, the only unknown yet is to check package manager.


#179

Wrong commit.
The right onet: https://github.com/troubadoour/sdwdate/commit/19503b03984370c040877a5b576e3d0a5a44e01c


#180

I don’t think it’s worth it to rewrite it. A lot unnecessary work. That code is very old, stable, needs little tweaking over the years. Shared with whonixcheck and tb-updater.

For debugging comment in “set -x” in /usr/lib/anon-shared-helper-scripts/te_pe_tb_check.

This is the problem.

+ rm --force '/tmp/tmp.dmqlVywOmc /tor_check_bootstrap_helper_bootstrap_file'

I.e. the environment variable ends with a new line. It should not.