Time Travel thanks to timesync

Development
1

Last time I found a bug I waited too long, and I promised to report it early on the next time. Today the timesync made the exact opposite of what it is supposed to do.

bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: Running sdwdate... bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: sdwdate_preparation: Setting CURL to curl.whonix-orig. bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching pre (SDW_MODE: daemon): /usr/lib/whonix/timesync_pre --autostart --mode $SDW_MODE bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching prerequisite (SDW_MODE: daemon) (CURL: curl.whonix-orig) (LD_PRELOAD: ): /usr/lib/whonix/timesync_prerequisite bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: DISPATCH_PREREQUISITE returned 0, continuing... bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: getUrlDateDiff: https://www.privacyinternational.org bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching SDWDATE_CURL_DISPATCH_PRE[SDWDATE_POOL_PAL] (SDW_MODE: daemon) (CURL: curl.whonix-orig): /usr/lib/whonix/msgcollector --icon /usr/share/whonix/icons/timesync.ico --identifier timesync --prog$ bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching SDWDATE_CURL_DISPATCH_POST[SDWDATE_POOL_PAL]: /usr/lib/whonix/msgcollector --icon /usr/share/whonix/icons/timesync.ico --identifier timesync --progressbaridx timesync --progressx 30 bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: https://www.privacyinternational.org (took 3s) => diff = 1 second(s) bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: getUrlDateDiff: https://thepiratebay.org bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching SDWDATE_CURL_DISPATCH_PRE[SDWDATE_POOL_NEUTRAL] (SDW_MODE: daemon) (CURL: curl.whonix-orig): /usr/lib/whonix/msgcollector --icon /usr/share/whonix/icons/timesync.ico --identifier timesync --$ bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching SDWDATE_CURL_DISPATCH_POST[SDWDATE_POOL_NEUTRAL]: /usr/lib/whonix/msgcollector --icon /usr/share/whonix/icons/timesync.ico --identifier timesync --progressbaridx timesync --progressx 60 bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: https://thepiratebay.org (took 3s) => diff = 25115 second(s) bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: getUrlDateDiff: https://encrypted.google.com bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching SDWDATE_CURL_DISPATCH_PRE[SDWDATE_POOL_FOE] (SDW_MODE: daemon) (CURL: curl.whonix-orig): /usr/lib/whonix/msgcollector --icon /usr/share/whonix/icons/timesync.ico --identifier timesync --prog$ bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching SDWDATE_CURL_DISPATCH_POST[SDWDATE_POOL_FOE]: /usr/lib/whonix/msgcollector --icon /usr/share/whonix/icons/timesync.ico --identifier timesync --progressbaridx timesync --progressx 80 bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: https://encrypted.google.com (took 3s) => diff = 2 second(s) bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: Median diff: 8372 second(s) bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: Setting time to 1398630421... bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: dispatching post_success (SDW_MODE: daemon): /usr/lib/whonix/timesync_post_success --autostart --mode $SDW_MODE bb9f72df-5e84-4dfb-bfaf-8e2cf4e2dce0: Sleeping for 33 minutes.

You can see of the three server the script asked, 2 of them agreed with the time of my machine roughly, but the pirate bay server is way off. And while the log says Median diff it actually calculates the average and it is applying this changed immediatly.

The average is: 8372 seconds, while the median would have been 2.

I heard you have a python developer now? :slight_smile: maybe rewriting this script into python would be a first good step. The error handling of Shell Scripts alone should be reason enough to avoid it for security critical applications like this one. And it would also help to make it more readable. Even with grepping I didnt find the method which should do the median calculation.

Is there a clean way to deactivate the automatic timesync? I currenty just return to skip the timesync. And is there a particular reason why whonix is not using ntp?

2

[quote=“whodam, post:1, topic:253”]You can see of the three server the script asked, 2 of them agreed with the time of my machine roughly, but the pirate bay server is way off. And while the log says Median diff it actually calculates the average and it is applying this changed immediatly.

The average is: 8372 seconds, while the median would have been 2.[/quote]
Current calculation (and my understanding of median as a non-native English speaker) is:
(1 + 25115 + 2) / 3 = ~8372

How would you calculate the median?

I am curious about suggestions on how to improve the algorithm.

Jason is working on a a thing we call “sneaky clock adjuster” for now (will probably get renamed before release). It gradually adjusts the clock, so results that are that wrong as in your unfortunate case won’t result in such an instant and big time travel [because sdwdate hopefully gets a better idea next time it runs]. Will most likely be included in Whonix 9. More info:
https://github.com/Whonix/Whonix/issues/169

I heard you have a python developer now? :) maybe rewriting this script into python would be a first good step.
Well, it's a volunteer python developer. I can request things on a polite "could you do that please" base, not on a employee "you ought to rewrite sdwdate until xxx" base. ;) This person is currently interested to work on gateway notice as well as on whonixcheck/timesync messages. Rewriting sdwdate would require a magnitude more time and skill. I don't know if this person is willing to take the time I am also not sure this is a good use of time.
The error handling of Shell Scripts alone should be reason enough to avoid it for security critical applications like this one.
What's wrong with the error handling?

I also understand how security would improve by using python. Attack surface currently is curl (in function sdwdate_download), the parsing code (in function sdwdate_parse_date) and date (in function sdwdate_convert_to_unixtime).

Even with grepping I didnt find the method which should do the median calculation.
In function sdwdate_build_median.
Is there a clean way to deactivate the automatic timesync?
Yes. We didn't have instructions for this, but I just finished writing some: https://www.whonix.org/wiki/Advanced_Security_Guide#Deactivate_Automatic_TimeSync
I currenty just return to skip the timesync.
That doesn't work. The interaction between sdwdate and timesync is currently poorly documented. And timesync is a poor name. Sdwdate is the real thing. Timesync is a monitor and GUI for sdwdate. Canceling timesync when started on boot as monitor (and GUI), won't terminate sdwdate. There is room for improvement and it's getting better as Whonix is split into multiple packages. (Started here: https://github.com/Whonix/sdwdate)
And is there a particular reason why whonix is not using ntp?
Yes. It doesn't work over Tor and it's not secure and impossible to be made secure as a distro redistributor [without patching ntp]. More info: - https://www.whonix.org/wiki/Dev/TimeSync#NTP_authentication_.28autokey.29_is_insecure - https://www.whonix.org/wiki/Dev/TimeSync
3
Well, it's a volunteer python developer. I can request things on a polite "could you do that please" base, not on a employee "you ought to rewrite sdwdate until xxx" base. ;) This person is currently interested to work on gateway notice as well as on whonixcheck/timesync messages. Rewriting sdwdate would require a magnitude more time and skill. I don't know if this person is willing to take the time I am also not sure this is a good use of time.

Of course, I’m open to any suggestions, it is just a matter of time, and of testing my limits in python coding. Remember that I am a defector from Windows :), and my programming experience comes from Pascal (Delphi) and .net (VB and C sharp.), where I have been used to strict variable declarations and strong type checking. Python seems reachable. When it comes to shell scripting, despite having the ABS guide handy, I am still at a loss. It’s a completely open way of coding that is still clashing with my habits.

Chiefly for the reason just mentioned, rewriting sdwdate in python is not in the pipeline, and it looks like an intricate piece. Let’s try to concentrate on msgcollector for the user interface.

I also understand how security would improve by using python. Attack surface currently is `curl` (in function sdwdate_download), the parsing code (in function sdwdate_parse_date) and `date` (in function sdwdate_convert_to_unixtime).

The AppArmor profile for curl has been ready for a while (it cannot be simpler), but I did not really test it yet. Coming soon. ‘date’ should not be a big problem.

4

In whatever language you are speaking normally it is probably called Median as well, what you calculate is the Average.

The median is defined as the element which splits a group of elements into two halfs of the same size, where in one half all elements are bigger or equal to the medien and in the other half are all elements which are smaller or equal to the median. Or in algorithmic terms, the set is sorted and you take the middle element. This helps to remove outlier. In statistics it is also equal to the 50-percentile.

Calculating the mediean in a set: [4, 8012, 9, 2,78] => sort => [2,4,9,78,8012] => middle element => 9, in contrast to the idea of the previous poster, this also works if the difference is big.

[quote]The error handling of Shell Scripts alone should be reason enough to avoid it for security critical applications like this one.[/quote] What's wrong with the error handling?

You make me really worry about the security aspect of Whonix if you ask things like that. Simply speaking: Shell script has no error handling worth calling it error handling.

Programming Language does matter

After Heartbleed we should be aware of that. In this case it is an algorithmic mistake though, and I still do not know how this should be in any way more safe than NTP. If a single server can change my time. At least take thepiratebay out of the loop, their time-base is wrong.

Yes. It doesn't work over Tor and it's not secure and impossible to be made secure as a distro redistributor [without patching ntp]. More info: - https://www.whonix.org/wiki/Dev/TimeSync#NTP_authentication_.28autokey.29_is_insecure - https://www.whonix.org/wiki/Dev/TimeSync
yeah thats what I feared that it is using UDP for latency reasons. However the security aspect is quite void after this bug. Thats exactly what I want to point out. Any of the time provider can currently change the clock of a clean installation of whonix. by any time he wants. How can ntp be ANY worse? There I can at least chose my providers.

I trust a tool like ntp much more than your script. thanks for your help in deactivating. The bug should seriously be fixed soon… jumping time is very bad for tor.

I am really puzzled how you look very paranoid on shared folders but do not care much about your coding (language, algorithms, coding standards) - I cannot help you though, debian always makes me cranky when working with their core, and yes that is BASH too… which is probably the reason why it makes me mad. Its actually really sad, I do not know any good introduction to best practices to make software secure. Maybe someone around has a secret tip.

@troubadour statically typed languages have certainly their advantages. My actual fear is, why is sdw-date such a big heap of code anyway. Timesychronisation ought to be very very simple.

It's a completely open way of coding that is still clashing with my habits.

Don’t lose that habit.

5

Having a AppArmor profile for curl/date is since. However, for sdwdate, I don’t think it’s needed. The sdwdate AppArmor profit restricts sdwdate. As far I understand, when it executes curl, that curl instance inherits from the sdwdate profile. So under the assumption, that sdwdate was compromised and AppArmor was safe, then sdwdate could not read any data in /home/user/something.

6

What you calculate exactly is the mean.

Quote from TimeSync: Whonix Time Synchronization Mechanism

It might make sense to add an external clock, such as a GPS, or even better an atomic clock. (Can we get an atomic USB clock?) This clock should be added to the host and/or Whonix-Gateway and/or Whonix-Workstation?

Open question: would the GPS/atomic clock be too accurate and would that make Whonix more fingerprintable?


I do not know if a cheap (~$20) GPS receiver is so accurate that it would make Whonix more fingerprintable, but from a quick look, it seems a feasible solution.

The gpsd daemon will read any GPS data format on an USB port, and most of the receivers speak NMEA, which is a standard. There are some python clients for gpsd around. It should be possible to integrate timesync in there. May be a lead to follow, as an option anyhow.

7

[quote=“whodam, post:4, topic:253”]In whatever language you are speaking normally it is probably called Median as well, what you calculate is the Average.

The median is defined as the element which splits a group of elements into two halfs of the same size, where in one half all elements are bigger or equal to the medien and in the other half are all elements which are smaller or equal to the median. Or in algorithmic terms, the set is sorted and you take the middle element. This helps to remove outlier. In statistics it is also equal to the 50-percentile.

Calculating the mediean in a set: [4, 8012, 9, 2,78] => sort => [2,4,9,78,8012] => middle element => 9, in contrast to the idea of the previous poster, this also works if the difference is big.[/quote]
I see. Would you recommend that for sdwdate?

[quote]The error handling of Shell Scripts alone should be reason enough to avoid it for security critical applications like this one.[/quote] What's wrong with the error handling?

You make me really worry about the security aspect of Whonix if you ask things like that. Simply speaking: Shell script has no error handling worth calling it error handling.

Programming Language does matter

After Heartbleed we should be aware of that. In this case it is an algorithmic mistake though,


Other question around. What kind of error should it handle that it does not handle?

I saw many of these claims, but never really something solid to back it up. The blog post you linked is speaking about the heartbleed bug. They used C. A lower level programming language. C is hard. Bash is a scripting language. Much simpler. The difficult parts of sdwdate are delegated to existing code (curl, bash).

and I still do not know how this should be in any way more safe than NTP.
NTP doesn't work over Tor (unless you add create a tcp tunnel capable of transporting udp beforehand), so it's out of question anyway. And with NTP, any man-in-the-middle can tamper with time requests, unless using authentication [which isn't safe, reference linked above], which isn't possible for a distro. sdwdate on the other hand uses 3 randomly selected servers [from pre-defined configurable lists] from three pools and builds the average.
If a single server can change my time.
This is indeed bad, but no one has come up with better suggestions yet. I've been saying this for so long. Reported bugs against Debian (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=%23687166), Ubuntu (https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1039420) and OpenBSD (http://thread.gmane.org/gmane.os.openbsd.bugs/18754). The only distros that are doing at least something about it are Tails (using tails_htp), Whonix (using sdwdate, which implements tails_htp in bash but uses the same algorithm). [And ChomeOS, that is using tlsdate?]
At least take thepiratebay out of the loop, their time-base is wrong.
Maybe. Updates take a while. When they are not fixing that.
There I can at least chose my providers.
You can change your providers in sdwdate as well. See /etc/sdwdate.d/30_sdwdate_default. It says: "## Please use "/etc/sdwdate.d/50_sdwdate_user" for your custom ## configuration, which will override the defaults found here." See SDWDATE_POOL_PAL, SDWDATE_POOL_NEUTRAL and SDWDATE_POOL_FOE for examples. Then add whatever servers your prefer to /etc/sdwdate.d/50_sdwdate_user.
I trust a tool like ntp much more than your script.
Unfortunately, it won't work over Tor.
The bug should seriously be fixed soon...
If you have any suggestions for the algorithm...
I am really puzzled how you look very paranoid on shared folders but do not care much about your coding (language, algorithms, coding standards)
No one could explain to me how bash worsens anything here. In a previous post I said what I believe what the attack surface is. I repeat and try to expand it a bit. Attack surface currently is `curl` (in function sdwdate_download), the parsing code (in function sdwdate_parse_date) and `date` (in function sdwdate_convert_to_unixtime).

Curl is written in C. Okay, that is bad. But is there a more secure/hardened downloader? When it was written in python, some python library would be used (or python core?). But python itself would interpret the code and python itself is written in C. How would that be any better? Has the python download code had more audits than curl?

As for the parsing code… The following: FIRST_WORD=“${LINE%%:*}” - uses untrusted input from the server. The server could prepare a line that abuses a remote code execution bug in bash’s C code if there is such a bug. How would python’s C code be any better for this job? And even if it is better, we could simply call python from bash to do this calculation. For bash I don’t see many CVE’s and none related to such string manipulation code.

debian always makes me cranky when working with their core, and yes that is BASH too... which is probably the reason why it makes me mad.
If you can tell me how for example /var/lib/dpkg/info/nano.postinst can worsens security just because it's written in bash... I never saw any serious argument about this either. What particular parts of Debian are written in bash and therefore worsening security?
My actual fear is, why is sdw-date such a big heap of code anyway.
I also don't agree with "lots of code = automatically lots of security issues".

sdwdate (the script) has 1112 lines of code in total. Up to line 429 [Whonix 8] it is just “irrelevant” for security. That’s almost 50%. There are variable defaults, reading config folder, parsing local command line options. How could the be exploited? Local exploit perhaps. But one is position to run such as local exploit wouldn’t require it anymore.

ntpdate.c alone has alone 2000 lines of C code. That’s just one code file among many files. And the ntp source code folder is much bigger. I would think that it is simpler to audit sdwdate than to review ntp.

The comparison sdwdate / ntpdate isn’t fair for many reasons. (Sdwdate fetches time from http headers while ntp does a lot more.) Also because sdwdate re-uses existing code for difficult tasks (delegates download to curl). Isn’t that what security is about? Re-using existing code to minimize the code base?

8

[quote=“troubadour, post:6, topic:253”]Quote from TimeSync: Whonix Time Synchronization Mechanism

I do not know if a cheap (~$20) GPS receiver is so accurate that it would make Whonix more fingerprintable, but from a quick look, it seems a feasible solution.

The gpsd daemon will read any GPS data format on an USB port, and most of the receivers speak NMEA, which is a standard. There are some python clients for gpsd around. It should be possible to integrate timesync in there. May be a lead to follow, as an option anyhow.[/quote]
GPS seems not that secure, since it depends on only one authoritative source.

What about an atomic clock? And before the amusement starts. This isn’t a joke. It’s not radioactive. I found one:

http://www.microsemi.com/products/timing-synchronization-systems/embedded-timing-solutions/components/sa-45s-chip-scale-atomic-clock

I haven’t found a price quote. A consumer class device, perhaps with USB would be required. That’s open for research if such a thing exists.

To which systems (hosts, gateway, workstation) do we feed it? Seems like this will only become a solution for advanced users. Install it only on the host, then disable sdwdate in all Whonix VMs, then just rely bootclockrandomization in Whonix VMs (in Whonix 9 enabled for Whonix-Gateway and Whonix-Workstation by default) to unlink host, gateway and workstation? That should perfectly solve pretty much all network time sync needs?

9

To be honest, I am not exactly sure what the intention of sdwdate was, I am not even quite sure what protocol it uses, but if you want to make it more fault tolerant it certainly would help. The median works a bit like a majority vote algorithm this helps to reduce the influence of any single time provider.

I saw many of these claims, but never really something solid to back it up. The blog post you linked is speaking about the heartbleed bug. They used C. A lower level programming language. C is hard. Bash is a scripting language. Much simpler. The difficult parts of sdwdate are delegated to existing code (curl, bash).

I disagree on the simplicity of Bash, it has a lot of special cases and exceptions to common rules. The handling of variables, when to put quotes or not, it bases on shell programs and has very little logic in itself. It is a language which makes it very simple to make something wrong. It is what it is, it is intended to automate processes you would do by hand. It works together very well with anything you would do in the Shell as well. It has been created to automate tasks and not for algorithms or complex programs. All you can do is checking return values, if you forget to handle it at one point that’s it, if the program will end gracefully will depend if the state is still somewhere where you expect it to be otherwise it will do something you did not expect (I recommend you to use at least set -eo pipefail ). In contrast ruby or python will throw or raise an exception which will go up to the top level if you forget to handle it. Of course you can cheat there too, but you will do it actively.

This is only one small example where modern programming languages protect you from yourself. I do not want to go beyond that.

This is an abstract discussion - and has less and less to do with the Problem at hand. I wanted to share my insight, in the end you will have to make the experience yourself. We all are the sum of our experiences. You will come to your own conclusions, I am not a genius that’s why I need languages which protect me and my costumers.

[quote]The bug should seriously be fixed soon...[/quote] If you have any suggestions for the algorithm...

Any form of majority vote will help, the median is certainly a good choice here. It depends how elaborate you wanna get. currently you ask 3 servers, so 1 of them can give wrong information with the median still being useful. If you sample from 5 it will even work if 2 servers will show the wrong time. Of course you could expand on this one, if some server is consistently disagreeing with other servers he could be temporarily be removed from the rotation. For a short term solution a median of 3 server will solve 99% of real world problems.

10

[quote=“whodam, post:9, topic:253”][quote]
Any form of majority vote will help, the median is certainly a good choice here. It depends how elaborate you wanna get. currently you ask 3 servers, so 1 of them can give wrong information with the median still being useful. If you sample from 5 it will even work if 2 servers will show the wrong time. Of course you could expand on this one, if some server is consistently disagreeing with other servers he could be temporarily be removed from the rotation. For a short term solution a median of 3 server will solve 99% of real world problems.
[/quote][/quote]

My vote. :wink:
The mean is a calculation used for “normal” distribution, that is when the samples are expected to deviate marginally from a given value. In our case, even if you take samples from, say, 10 servers, the value returned by thepiratebay would not be smoothed out by the mean. It is taken out de facto by using the median.

11

Using the average really is a bad idea. Fixed to median (fix, use median instead of average as suggested in https://www.whonix… · Kicksecure/sdwdate@4826248 · GitHub). Will appear in Whonix 9.