sdwdate and sdwdate-gui development thread

Development
41

Yes, this is expected. When files are removed from the package and the package is reinstalled, then old files are removed. (With exceptions, /etc.)
The solution is to have these files in your branch in the correct location. Then next time they will be [re]installed. Seems like you did that already.

42

Please restore:
debian/sdwdate.postinst
debian/sdwdate.postrm

Not sure the move of sdwdate to /usr/sbin is good. Maybe /usr/bin is better. Maybe some day we want to support a oneshot, non-daemon operation mode? I.e. type “sdwdate” -> replies with time.

43

[quote=“troubadour, post:30, topic:1137”]- Made all the pool members “single line”.
https://github.com/troubadoour/sdwdate/commit/b3520602181e368ae563182b8b1c7032dc044d08[/quote]
I think this was a misunderstanding. Now previously grouped pool members, are not a group anymore. Therefore those members are chosen much more often. Thereby get more influence.

Not good:
meld etc/sdwdate.d/30_sdwdate_default etc/sdwdate-python.d/30_sdwdate_default

What I meant by ‘Make all pool members “multi lined”’…

Original sdwdate-bash:

SDWDATE_POOL_TWO=(
         "
            atlas777hhh7mcs7.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            compass6vpxj32p3.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            globe223ezvh6bps.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            bbbbbb6qtmqg65g6.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            pppppptkftqqnfsq.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
         "
         "w6csjytbrl273che.onion#Ljost[24][25] 	2012-September-30 	Transparency Activism 	w6csjytbrl273che.onion 	https://w6csjytbrl273che.tor2web.org/ 	Iceland"
         "ak2uqfavwgmjrvtu.onion#MagyarLeaks[26] 	2013-July-7 	Investigative Journalism 	ak2uqfavwgmjrvtu.onion 	https://ak2uqfavwgmjrvtu.tor2web.org 	Hungary"
         ...
)

Proposed if that helps parsing:

SDWDATE_POOL_TWO=(
         "
            atlas777hhh7mcs7.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            compass6vpxj32p3.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            globe223ezvh6bps.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            bbbbbb6qtmqg65g6.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
            pppppptkftqqnfsq.onion:80#Hosted by Thomas White. https://www.whonix.org/pipermail/whonix-devel/2015-February/000297.html
         "
         "
            w6csjytbrl273che.onion#Ljost[24][25] 	2012-September-30 	Transparency Activism 	w6csjytbrl273che.onion 	https://w6csjytbrl273che.tor2web.org/ 	Iceland
         "
         "
            ak2uqfavwgmjrvtu.onion#MagyarLeaks[26] 	2013-July-7 	Investigative Journalism 	ak2uqfavwgmjrvtu.onion 	https://ak2uqfavwgmjrvtu.tor2web.org 	Hungary
         "
         ...
)

In other words, from the initial random picking of a pool member “Thomas White”, a multi lined mirror, i.e. a pool member providing mirrors should only appear as one. Not many. And upon picking such a multi lined pool member “Thomas White”, some random mirror he is providing gets chosen.

44

typo: eaxh

                if len(self.urls) == 0:
                    ## Most likely, internet connection is down.
                    ## Raise eror, log.
                    message = ('No values returned from url_to_unixtime. Internet connection might be down.')
                    print(message)
                    logger.critical(message)
                    sys.exit()

Please don’t exit. The daemon must be able to cope up in case internet connection is down.

    def general_proxy_error(self, pools):
        '''
        This error occurs (at least) when Tor is not running.
        '''
        if (pools[0] == 'Connection closed unexpectedly' and
            pools[1] == 'Connection closed unexpectedly' and
            pools[2] == 'Connection closed unexpectedly'):
                ## Raise error, log, user warning.
                logger.critical('General Proxy Error')
                sys.exit(1)

Also no reason to exit.

The right fall back action in such special cases is falling back to the loop (which will then continue with a randomized sleep).

45

Bug: If set_new_time returns false, because median equals zero, then the first_success_file does not get created. However, such cases can be viewed as sdwdate having run successful. first_success_file will be used as an indicator by applications (gui) using state information provided by sdwdate.

46

[quote=“Patrick, post:42, topic:1137”]Please restore:
debian/sdwdate.postinst
debian/sdwdate.postrm[/quote]

Not sure the move of sdwdate to /usr/sbin is good. Maybe /usr/bin is better. Maybe some day we want to support a oneshot, non-daemon operation mode? I.e. type "sdwdate" -> replies with time.
Yes, moved it to /usr/bin. There are some minor changes in sdwdate that I forgot to commit earlier. https://github.com/troubadoour/sdwdate/commit/2142fd9a22cf4e857090d13c617465b205d083f6

Restored the multi-line entries (groups) in 30_sdwdate_default. The new config.py can read any number of groups. It appends one random member from each group to the pool list.
For consistency, the groups are separated by a line with one double-quote (could be anything), but the members have the same format as the single entries, starting with a double-quote. That lets us use the same regular expression operation for parsing the urls from single or multi entries.

The sys.exit() are only temporary solutions. Have yet to implement the error handling.

Bug: If set_new_time returns false, because median equals zero, then the first_success_file does not get created. However, such cases can be viewed as sdwdate having run successful. first_success_file will be used as an indicator by applications (gui) using state information provided by sdwdate.
Yes. Will modify that.
47

There is still a diff. Some stuff that I don’t understand.

kdiff3 etc/sdwdate.d/30_sdwdate_default etc/sdwdate-python.d/30_sdwdate_default

Please check.

Experimenting with a few inline comments:

48

The inline comments are quite useful.

I’ve been making some experiments with etc/sdwdate-python.d/30_sdwdate_default, hence the readability issue.

Pushed a new one which should better show the differences.

  • replaced the extraneous quote by square brackets. For parsing the groups, a marker is required, anyhow [the modified config.py is in a separate commit].
  • added the missing quotes.
  • some entries are redundant, or misspelled (missing .onion). Removed or edited.
  • the parsing script needs a consistent format for the urls, ex. "cwoiopiifrlzcuos.onion#. The ones with the port extension :80 are edited.
49

Pushed two commits:

  • error handling.
  • test .d configuration files with invalid pools.

There is error logging only so far, because I’m trying to find the way to pass messages from sdwdate to the timesync systray icon, which is running its own event loop. Some progress…

50

Moved the test configuration files to /usr/share, because they stop sdwdate (to be investigated).

51

Yes, we haven’t thought through yet how sdwdate would communicate with the tray icon. I think RPC can and should be avoided for simplicity. The tray icon should omit minor details. For example, if fetching some onion failed but time synchronization still succeeded, this information can be buried in sdwdate’s log.

I think ‘Sdwdate Connection Checker Plugin’ - https://github.com/Whonix/sdwdate-plugin-anon-shared-con-check - no longer needs to be a plugin. Because sdwdate depends on Tor now so or so. Not sure if it would still be useful to have the code that determines the status of connectivity would be configurable / replaceable by custom code (setting).

I made a first sketch how this interaction could work:

Please feel free to make bold changes. This needs some hard criticism, so sdwdate-tray can provide good usability.

52

There is a sdwdate-tray template in GitHub - troubadoour/timesync, python branch.

The approach is simple: besides the Qt event loop, sdwdate_tray runs a second thread, polling a status file updated by sdwdate daemon. It thus acts as a plain (nearly) real time sdwdate monitor, As development goes, I guess it should be able to perform two actions: release or restrict the network.

Have updated the wiki. TimeSync: Whonix ™ Time Synchronization Mechanism

53

first version by me:

changes by troubadour:

second version by troubadour:

I very much like these changes. Much simpler.

We should avoid /tmp though. Files in /tmp are not supposed to be shared files as far I know. /var/run/sdwdate seems more appropriate. Or /var/lib/sdwdate. Actually, I think the latter is more appropriate. But /var/run/sdwdate is better, because it’s empty after reboot. (Guarding against cases where sdwdate-tray starts before sdwdate-daemon. In these cases it should not be fed by outdated information.)

More comments to come.

54

I think the timesync repository is the wrong place for sdwdate-tray.

  • Perhaps sdwdate-tray -> sdwdate-gui?
  • repository name: sdwdate-gui?
  • timesync was a bad name chosen by me earlier. Aids confusing the functionality of sdwdate with timesync. Seems like two competitors.
  • timesync provides a cli interface to sdwdate (monitor / restart).
  • As of Whonix 10 / 11, timesync gets called by sdwdate hooks. It informs cli users about the status of network time synchronization in tty0.
  • I am not sure about that functionality. Maybe it should be deprecated.
55
  • restrict: in which cases that would be useful? I think restricting the network after boot is better implemented in Whonix’s firewall.
  • release: not directly → dispatching the [first] success hook would care of this to keep sdwdate and whonix’s firewall separated
56
polling a file
Polling the file system isn't good. On some systems /var/run isn't mounted in RAM. Therefore causing an infinite stream of disk reads. It's better to read the file event based. When notified by the kernel. That's what inotifywait is for.
57

Maybe we can have two files maintained by sdwdate-daemon, read by sdwdate-tray…

  • /var/run/sdwdate/tray.color
  • /var/run/sdwdate/tray.message
    What do you think?
58

Changes by me:

  • Added ideas for sdwdate-tray left click action.
  • Edited normal cycle. No popups. Those are super intrusive and annoying. When running multiple VMs and having an issue, that’s x popups.
[done-success] could be handled in the messaging mechanism (not that many possible combinations).
done/success files are useful for other applications checking sdwdate's status. Useful for use in scripts. And for manual investigation it's nice to just check the status file.
The asynchronous times fetching in sdwdate-daemon should render the progress feature obsolete. Yet a safety could be implemented in sdwdate-daemon for the case url_to_unixtime fails.
Due to the asynchronous time fetching... And due to the non-availability of progress during connection initiation from python libs and other downloaders... [Those can only provide a download speed indicator. But what we transmit is super small. And only takes only the smallest time. DNS resolution and initial connection setup cannot be measured as far as I know.] Therefore a progress in percentage will indeed no longer be possible.

However, having a status “progress” is still important. Network can be super slow at times. [Flaky mobile network connection, GPRS, throttling and other stuff.] Even though the asynchronous time fetching may be super fast on our developer machines, it must be expected to be a lot slower on other systems. Taking a minute or even longer. Therefore a color and message for “progress” is required.

(2) In most of the cases, time synchronization should be completed before sdwdate-tray is ready (kdm started, X running).
True. But please also expect/text the corner cases. With a distribution at this size, we're guaranteed to hit those.
sdwdate-tray starts after X is running. Checks bootclockrandomization, checks sdwdate (1), reads the last sdwdate-daemon status file, removes the file.
Removing the file is probably not good. Makes this more obscure. More difficult to debug. In case it's suspected sdwdate-tray shows a wrong message, one couldn't check if the cause was sdwdate-daemon providing a wrong message. While just keeping overwriting the file, we can ask users to manually con`cat`enate it.

I was always wondering if timesync (Whonix 10 / 11) checking the status of bootclockrandomization was a design mistake. Perhaps that rather should be a feature of sdwdate? If bootclockrandomization is installed, check it. Otherwise, skip it.

Same for timesanitycheck. If timesanitycheck is installed, check it. Otherwise, skip it.

Failed bootclockrandomization and/or timesanitycheck would influence the tray icon color and hovering over message.

What do you think?

59
I think the timesync repository is the wrong place for sdwdate-tray. - Perhaps sdwdate-tray -> sdwdate-gui? - repository name: sdwdate-gui?
timesync has always been confusing for me too. And a lot of files in timesync will not be used. So, let's say a new sdwdate-gui repository.
- restrict: in which cases that would be useful? I think restricting the network after boot is better implemented in Whonix's firewall. - release: not directly -> dispatching the [first] success hook would care of this to keep sdwdate and whonix's firewall separated
OK,
Polling the file system isn't good. On some systems /var/run isn't mounted in RAM. Therefore causing an infinite stream of disk reads. It's better to read the file event based. When notified by the kernel. That's what inotifywait is for.
The tests with a file in @HOME (without removing it) did not show any performance penalty, and we can increase time.sleep() in the loop. But it's probably not the best approach. Will try inotify.
Maybe we can have two files maintained by sdwdate-daemon, read by sdwdate-tray... - /var/run/sdwdate/tray.color - /var/run/sdwdate/tray.message What do you think?
Yes, easier to manage, less code for discriminating the errors.
However, having a status "progress" is still important. Network can be super slow at times. [Flaky mobile network connection, GPRS, throttling and other stuff.] Even though the asynchronous time fetching may be super fast on our developer machines, it must be expected to be a lot slower on other systems. Taking a minute or even longer. Therefore a color and message for "progress" is required.
True. But please also expect/text the corner cases. With a distribution at this size, we're guaranteed to hit those.
In case super slow network, the user would be notified by the color of the icon (status reporting "Fetching time from remotes" or something like that).
Removing the file is probably not good. Makes this more obscure. More difficult to debug. In case it's suspected sdwdate-tray shows a wrong message, one couldn't check if the cause was sdwdate-daemon providing a wrong message. While just keeping overwriting the file, we can ask users to manually con`cat`enate it.
If we use inotify, sdwdate can overwrite the file, and sdwdate-gui only read it without removing it.
I was always wondering if timesync (Whonix 10 / 11) checking the status of bootclockrandomization was a design mistake. Perhaps that rather should be a feature of sdwdate? If bootclockrandomization is installed, check it. Otherwise, skip it.
I was going to ask why bootclockrandomization check was not done in sdwdate. That would be the logical place to implement it, as well as timesanitycheck. Skip if not existing, report error - require color change in case of failure of either or both.
60

Great stuff.

inotifywait → My comment were about sdwdate-gui. Because reading /var/run/sdwdate/[…] files every or every few seconds is too much disk i/o. In other words, if sdwdate-gui would be a loop that reads the status files every few seconds, that would be too much disk i/o.

sdwdate-daemon probably does not require inotifywait. It can do normal writes. Those are infrequent enough. That’s fine.

In case super slow network, the user would be notified by the color of the icon (status reporting "Fetching time from remotes" or something like that).
Yes. Color: - (a) on boot / when first success file does not exist: yellow - (b) during normal cycle: stay green (no need to make the user crazy by regularly switching color) - (c) on manual restart: yellow [same case as (a)?] (user expects color change then, active action]

(I am open to whether yellow is the best color here. Maybe one has an argument for blue or otherwise.)

If we use inotify, sdwdate can overwrite the file, and sdwdate-gui only read it without removing it.
Yes, that's what I had in mind. (There is a "modify" event that inotifywait can dispatch. Happens on overwrite.)
I was going to ask why bootclockrandomization check was not done in sdwdate. That would be the logical place to implement it, as well as timesanitycheck. Skip if not existing, report error - require color change in case of failure of either or both.
Yes. Please question such strange things.

By the way, in case anyone is wondering why bootclockrandomization action is not in sdwdate: Because not everyone might want to use that. More interesting in case of anonymity distributions. Therefore keeping sdwdate fit for inclusion into upstream repositories such as Debian. There is interest in NTP replacement tools. (Don't update NTP – stop using it - Hanno's blog) And I think going more mainstream would be a good thing. Therefore I am keeping the non-Whonix use case in mind. After all, one day we also need secure network time on [non-anonymous] hosts.