sdwdate and sdwdate-gui development thread

The inline comments are quite useful.

I’ve been making some experiments with etc/sdwdate-python.d/30_sdwdate_default, hence the readability issue.

Pushed a new one which should better show the differences.

  • replaced the extraneous quote by square brackets. For parsing the groups, a marker is required, anyhow [the modified config.py is in a separate commit].
  • added the missing quotes.
  • some entries are redundant, or misspelled (missing .onion). Removed or edited.
  • the parsing script needs a consistent format for the urls, ex. "cwoiopiifrlzcuos.onion#. The ones with the port extension :80 are edited.

Pushed two commits:

  • error handling.
  • test .d configuration files with invalid pools.

There is error logging only so far, because I’m trying to find the way to pass messages from sdwdate to the timesync systray icon, which is running its own event loop. Some progress…

Moved the test configuration files to /usr/share, because they stop sdwdate (to be investigated).

Yes, we haven’t thought through yet how sdwdate would communicate with the tray icon. I think RPC can and should be avoided for simplicity. The tray icon should omit minor details. For example, if fetching some onion failed but time synchronization still succeeded, this information can be buried in sdwdate’s log.

I think ‘Sdwdate Connection Checker Plugin’ - https://github.com/Whonix/sdwdate-plugin-anon-shared-con-check - no longer needs to be a plugin. Because sdwdate depends on Tor now so or so. Not sure if it would still be useful to have the code that determines the status of connectivity would be configurable / replaceable by custom code (setting).

I made a first sketch how this interaction could work:

Please feel free to make bold changes. This needs some hard criticism, so sdwdate-tray can provide good usability.

There is a sdwdate-tray template in GitHub - troubadoour/timesync, python branch.

The approach is simple: besides the Qt event loop, sdwdate_tray runs a second thread, polling a status file updated by sdwdate daemon. It thus acts as a plain (nearly) real time sdwdate monitor, As development goes, I guess it should be able to perform two actions: release or restrict the network.

Have updated the wiki. TimeSync: Whonix ™ Time Synchronization Mechanism

first version by me:

changes by troubadour:

second version by troubadour:

I very much like these changes. Much simpler.

We should avoid /tmp though. Files in /tmp are not supposed to be shared files as far I know. /var/run/sdwdate seems more appropriate. Or /var/lib/sdwdate. Actually, I think the latter is more appropriate. But /var/run/sdwdate is better, because it’s empty after reboot. (Guarding against cases where sdwdate-tray starts before sdwdate-daemon. In these cases it should not be fed by outdated information.)

More comments to come.

I think the timesync repository is the wrong place for sdwdate-tray.

  • Perhaps sdwdate-tray -> sdwdate-gui?
  • repository name: sdwdate-gui?
  • timesync was a bad name chosen by me earlier. Aids confusing the functionality of sdwdate with timesync. Seems like two competitors.
  • timesync provides a cli interface to sdwdate (monitor / restart).
  • As of Whonix 10 / 11, timesync gets called by sdwdate hooks. It informs cli users about the status of network time synchronization in tty0.
  • I am not sure about that functionality. Maybe it should be deprecated.
  • restrict: in which cases that would be useful? I think restricting the network after boot is better implemented in Whonix’s firewall.
  • release: not directly → dispatching the [first] success hook would care of this to keep sdwdate and whonix’s firewall separated
polling a file
Polling the file system isn't good. On some systems /var/run isn't mounted in RAM. Therefore causing an infinite stream of disk reads. It's better to read the file event based. When notified by the kernel. That's what inotifywait is for.

Maybe we can have two files maintained by sdwdate-daemon, read by sdwdate-tray…

  • /var/run/sdwdate/tray.color
  • /var/run/sdwdate/tray.message
    What do you think?

Changes by me:

  • Added ideas for sdwdate-tray left click action.
  • Edited normal cycle. No popups. Those are super intrusive and annoying. When running multiple VMs and having an issue, that’s x popups.
[done-success] could be handled in the messaging mechanism (not that many possible combinations).
done/success files are useful for other applications checking sdwdate's status. Useful for use in scripts. And for manual investigation it's nice to just check the status file.
The asynchronous times fetching in sdwdate-daemon should render the progress feature obsolete. Yet a safety could be implemented in sdwdate-daemon for the case url_to_unixtime fails.
Due to the asynchronous time fetching... And due to the non-availability of progress during connection initiation from python libs and other downloaders... [Those can only provide a download speed indicator. But what we transmit is super small. And only takes only the smallest time. DNS resolution and initial connection setup cannot be measured as far as I know.] Therefore a progress in percentage will indeed no longer be possible.

However, having a status “progress” is still important. Network can be super slow at times. [Flaky mobile network connection, GPRS, throttling and other stuff.] Even though the asynchronous time fetching may be super fast on our developer machines, it must be expected to be a lot slower on other systems. Taking a minute or even longer. Therefore a color and message for “progress” is required.

(2) In most of the cases, time synchronization should be completed before sdwdate-tray is ready (kdm started, X running).
True. But please also expect/text the corner cases. With a distribution at this size, we're guaranteed to hit those.
sdwdate-tray starts after X is running. Checks bootclockrandomization, checks sdwdate (1), reads the last sdwdate-daemon status file, removes the file.
Removing the file is probably not good. Makes this more obscure. More difficult to debug. In case it's suspected sdwdate-tray shows a wrong message, one couldn't check if the cause was sdwdate-daemon providing a wrong message. While just keeping overwriting the file, we can ask users to manually con`cat`enate it.

I was always wondering if timesync (Whonix 10 / 11) checking the status of bootclockrandomization was a design mistake. Perhaps that rather should be a feature of sdwdate? If bootclockrandomization is installed, check it. Otherwise, skip it.

Same for timesanitycheck. If timesanitycheck is installed, check it. Otherwise, skip it.

Failed bootclockrandomization and/or timesanitycheck would influence the tray icon color and hovering over message.

What do you think?

I think the timesync repository is the wrong place for sdwdate-tray. - Perhaps sdwdate-tray -> sdwdate-gui? - repository name: sdwdate-gui?
timesync has always been confusing for me too. And a lot of files in timesync will not be used. So, let's say a new sdwdate-gui repository.
- restrict: in which cases that would be useful? I think restricting the network after boot is better implemented in Whonix's firewall. - release: not directly -> dispatching the [first] success hook would care of this to keep sdwdate and whonix's firewall separated
OK,
Polling the file system isn't good. On some systems /var/run isn't mounted in RAM. Therefore causing an infinite stream of disk reads. It's better to read the file event based. When notified by the kernel. That's what inotifywait is for.
The tests with a file in @HOME (without removing it) did not show any performance penalty, and we can increase time.sleep() in the loop. But it's probably not the best approach. Will try inotify.
Maybe we can have two files maintained by sdwdate-daemon, read by sdwdate-tray... - /var/run/sdwdate/tray.color - /var/run/sdwdate/tray.message What do you think?
Yes, easier to manage, less code for discriminating the errors.
However, having a status "progress" is still important. Network can be super slow at times. [Flaky mobile network connection, GPRS, throttling and other stuff.] Even though the asynchronous time fetching may be super fast on our developer machines, it must be expected to be a lot slower on other systems. Taking a minute or even longer. Therefore a color and message for "progress" is required.
True. But please also expect/text the corner cases. With a distribution at this size, we're guaranteed to hit those.
In case super slow network, the user would be notified by the color of the icon (status reporting "Fetching time from remotes" or something like that).
Removing the file is probably not good. Makes this more obscure. More difficult to debug. In case it's suspected sdwdate-tray shows a wrong message, one couldn't check if the cause was sdwdate-daemon providing a wrong message. While just keeping overwriting the file, we can ask users to manually con`cat`enate it.
If we use inotify, sdwdate can overwrite the file, and sdwdate-gui only read it without removing it.
I was always wondering if timesync (Whonix 10 / 11) checking the status of bootclockrandomization was a design mistake. Perhaps that rather should be a feature of sdwdate? If bootclockrandomization is installed, check it. Otherwise, skip it.
I was going to ask why bootclockrandomization check was not done in sdwdate. That would be the logical place to implement it, as well as timesanitycheck. Skip if not existing, report error - require color change in case of failure of either or both.

Great stuff.

inotifywait → My comment were about sdwdate-gui. Because reading /var/run/sdwdate/[…] files every or every few seconds is too much disk i/o. In other words, if sdwdate-gui would be a loop that reads the status files every few seconds, that would be too much disk i/o.

sdwdate-daemon probably does not require inotifywait. It can do normal writes. Those are infrequent enough. That’s fine.

In case super slow network, the user would be notified by the color of the icon (status reporting "Fetching time from remotes" or something like that).
Yes. Color: - (a) on boot / when first success file does not exist: yellow - (b) during normal cycle: stay green (no need to make the user crazy by regularly switching color) - (c) on manual restart: yellow [same case as (a)?] (user expects color change then, active action]

(I am open to whether yellow is the best color here. Maybe one has an argument for blue or otherwise.)

If we use inotify, sdwdate can overwrite the file, and sdwdate-gui only read it without removing it.
Yes, that's what I had in mind. (There is a "modify" event that inotifywait can dispatch. Happens on overwrite.)
I was going to ask why bootclockrandomization check was not done in sdwdate. That would be the logical place to implement it, as well as timesanitycheck. Skip if not existing, report error - require color change in case of failure of either or both.
Yes. Please question such strange things.

By the way, in case anyone is wondering why bootclockrandomization action is not in sdwdate: Because not everyone might want to use that. More interesting in case of anonymity distributions. Therefore keeping sdwdate fit for inclusion into upstream repositories such as Debian. There is interest in NTP replacement tools. (Don't update NTP – stop using it - Hanno's blog) And I think going more mainstream would be a good thing. Therefore I am keeping the non-Whonix use case in mind. After all, one day we also need secure network time on [non-anonymous] hosts.

sdwdate-daemon probably does not require inotifywait. It can do normal writes. Those are infrequent enough. That's fine.
I was meaning inotify in sdwdate-gui.

Pushed new sdwdate-gui packages. GitHub - troubadoour/sdwdate-gui.

As usual, a couple of lintian warnings.
There must be a way around “new-package-should-close-itp-bug”. I won’t file a bug in Debian, or add “Close #123456”.
Tried to override “zero-byte-file-in-doc-directory” without luck.

There must be a way around "new-package-should-close-itp-bug". I won't file a bug in Debian, or add "Close #123456".
No need. It will go away with the next version number. Just bump the version number. When the second changelog entry exists, it won't complain about this anymore.

Handy shortcut.

make deb-chl-bumpup
Tried to override "zero-byte-file-in-doc-directory" without luck.
Will look into this now.

No need to override. It’s because changelog.upstream is still empty. deb-chl-bumpup will also fix this.

export DEBFULLNAME="Full Name"
export DEBEMAIL=your@emai.net
make deb-chl-bumpup

(You could set this environment variables more comfortably in a location of your choice to make those permanent.)
(deb-chl-bumpup is just automating and imho simplifying usage of debchange.)

Updated the wiki.

Package name. After your mail to Whonix-devel and the name naturally used in TimeSync: Whonix ™ Time Synchronization Mechanism, I think sdwdate-tray is more descriptive than sdwdate-gui, which sounds too general. I propose to rename the package sdwdate-tray.

Not sure. How likely is it that sdwdate-tray turns into a full gui? Especially the left click action could show a comprehensive overview of all states of the various checks.

How likely is it that sdwdate-tray turns into a full gui?
Quite likely. So we keep sdwdate-gui.

Two commits:

Using QFileSystemWatcher in sdwdate-gui for monitoring sdwdate status file.

A few remarks:

  • the Exit menu entry in sdwdate-gui context menu is for testing. Will be removed.
  • a single status file: python objects are serialized in sdwdate and deserialized in sdwdate-gui. We could put anything in there…
  • the icons are downloaded from mediawiki, for demonstration.
  • three icons at this stage: success, busy, error.

More to come. Minor and perhaps less minor debugging, and some questions.