There is a modified standalone url_to_unixtime at GitHub - troubadoour/url_to_unixtime-asynchronous.
The idea is to get the time from the pool’s remotes asynchronously, using some of the Python gevent magic. Besides an improved transparency of timesync, the rationale is that if we want to implement ⚓ T300 replace timesync progress bar by tray icon sdwdate-gui and block internet access after boot as long as the time synchronization has not finished, the shorter it takes, the better for the user.
In the script, for the sake of demonstration, the three pools are concatenated into a single one. After running it a sufficient number of times, you should have seen many combinations of errors (connect error, timeout, header parsing…). In all the cases, the duration of the acquisition is reduced drastically.
I do not know if this approach looks sound, but it would require some major changes in sdwdate. The pools being moved in url_to_unixtime, all the logic from sdwdate_pick and above (sdwdate_loop, sdwdate_get_time_from_remote and even sdwdate_build_median) would be done in the new module(s). Perhaps a starting point to port sdwdate to Python.
Btw the pool members are outdated. Check bottom of /etc/sdwdate.d/30_sdwdate_default.
A separate branch in the sdwdate repository might be sufficient/better.
Maybe we should keep url_to_unixtime a separate tool as is? Would make it easier to contain security wise? The time required to fork a child process should be negligible?
The question is how many tasks would remain in sdwdate? Perhaps while doing these extensive changes, we could abolish the remaining bash parts. A few features that we don’t use could be removed from sdwdate. Some thoughts…
-
drop environment variable support
-
drop command line parsing support
-
perhaps drop SYSTOHTC support
-
drop DEBUG=, USER=, DONT_SET_DATE=
-
drop SDWDATE_SCLOCKADJ_* → hardcode
-
drop ECHO_UNIX_TIME
-
drop --quiet, --cache-dir, --temp-dir
-
drop --(done|success|first_success)_file support (just create those, but not configurable paths)
-
drop built-in (hardcoded) pools, just refuse to work if config file does not exist?
-
config folder parsing only
-
keep the pool, retry, ALLOWED_PER_POOL_FAILURE_RATIO design
-
keep unit test that uses common functions to check all the hidden services for reachability that devs can run every now and then to see which should be removed
Should we keep the current plugin / dispatcher structure?
current plugins:
The idea is to have a strong separation between anonymity distribution vs regular distribution as well as a strong separation between terminal and gui parts. And therefore to keep sdwdate simpler. Also keeping sclockadj out of it.
We could go through sdwdate functions and features one by one. See what we use now, what we require in future. Then sketch a rough high level overview how new python functions could be called and what they should basically do.
Should we create a wiki page for that sketch?
Yes.
Maybe we should keep url_to_unixtime a separate tool as is? Would make it easier to contain security wise? The time required to fork a child process should be negligible?
Yes, in its own module, reading its configuration from /etc/sdwdate.d.
The question is how many tasks would remain in sdwdate? Perhaps while doing these extensive changes, we could abolish the remaining bash parts. A few features that we don't use could be removed from sdwdate. Some thoughts...
- drop environment variable support
- drop command line parsing support
- perhaps drop SYSTOHTC support
- drop DEBUG=, USER=, DONT_SET_DATE=
- drop SDWDATE_SCLOCKADJ_* → hardcode
- drop ECHO_UNIX_TIME
- drop --quiet, --cache-dir, --temp-dir
- drop --(done|success|first_success)_file support (just create those, but not configurable paths)
That would make sense, and life easier
- config folder parsing only
- keep the pool, retry, ALLOWED_PER_POOL_FAILURE_RATIO design
- keep unit test that uses common functions to check all the hidden services for reachability that devs can run every now and then to see which should be removed
Yes. The unit test could be as simple as running url_to_unixtime with an argument "--sdout" or something, outputting the values to the terminal instead of returning them to the calling module.
Should we keep the current plugin / dispatcher structure?
current plugins:
Probably keep them, still early to tell.
- https://github.com/Whonix/timesync : gui
Probably keep it for the time being. It will become obsolete at some stage of the development, but I'm not familiar enough yet with the intricacies of timesync / sdwdate to be able tot tell when.
We could go through sdwdate functions and features one by one. See what we use now, what we require in future. Then sketch a rough high level overview how new python functions could be called and what they should basically do.
Sounds good.
Should we create a wiki page for that sketch?
Do you mean we would discuss the details in the forum, and outline the results in the wiki?
[quote=“troubadour, post:3, topic:1137”][quote]
Maybe we should keep url_to_unixtime a separate tool as is? Would make it easier to contain security wise? The time required to fork a child process should be negligible?
[/quote]
Yes, in its own module, reading its configuration from /etc/sdwdate.d.[/quote]
Minor…
I recommend to keep config reading and calling to url_to_unixtime to sdwdate(-python). [I don’t want to imply a new package name, just rewrite of sdwdate-bash to sdwdate-python.)
url_to_unixtime would just be the url to unixtime tool. That way file access and whatnot could be limited to a minimum.
Yes. The unit test could be as simple as running url_to_unixtime with an argument "--sdout" or something, outputting the values to the terminal instead of returning them to the calling module.
Yes.
Do you mean we would discuss the details in the forum, and outline the results in the wiki?
Yes. It's just an idea. Making a high level overview in human speech what functions such as sdwdate_build_median are doing, what other functions etc.
I recommend to keep config reading and calling to url_to_unixtime to sdwdate(-python). [I don't want to imply a new package name, just rewrite of sdwdate-bash to sdwdate-python.)
url_to_unixtime would just be the url to unixtime tool. That way file access and whatnot could be limited to a minimum.
Yes, I was still in sdwdate-bash, where I would not know how to pass a list.
[quote]Do you mean we would discuss the details in the forum, and outline the results in the wiki?[/quote]
Yes. It's just an idea. Making a high level overview in human speech what functions such as sdwdate_build_median are doing, what other functions etc.
Good idea. It might help us clarifying our work too.
A couple of questions to start with.
-
Do we use use only onion sites, or we must be able to revert to clearnet sites?
-
Do we want to daemonize sdwdate the same way we did in control-port-filter=python, which finally is declared as service (not fork) in the systemd unit file? It seems overkill because systemd takes care of the daemonization, and python-daemon looks redundant. A “while true” loop could do the job. There is sd_notify … have not looked into it yet.
- Do we use use only onion sites, or we must be able to revert to clearnet sites?
I hope we don't have to ever revert to clearnet with ssl insecurities and such.
Do we want to daemonize sdwdate the same way we did in control-port-filter=python, which finally is declared as service (not fork) in the systemd unit file? It seems overkill because systemd takes care of the daemonization, and python-daemon looks redundant. A "while true" loop could do the job. There is sd_notify ... have not looked into it yet.
The simpler way without python-daemon looks fine to me.
Not sure what you mean. Added the onion sites, hardcoded in the script, from the pools in 30_sdwdate_ default.
https://github.com/troubadoour/url_to_unixtime-asynchronous/commit/cb5faa8a0de29e8a42467eb2f1595bcbafb5dd2c
Will start on the configuration files. May be a separate file for the pools?
The provisional plan:
- define the format for the pools file.
- write / test the parsing function in url_to_unixtime.
- move it in a separate module, because it should be used by both sdwdate and url_to_unixtime in standalone mode (check the sites in the pools).
In other words…
Drop:
Keep:
[hr]
Will start on the configuration files. May be a separate file for the pools?
I would like to keep it in one. .d-style parsing as with cpfpy.
- define the format for the pools file.
The current pool syntax is.
"url.onion[:port]#comment"
"
url.onion[:port]#comment
[url.onion[:port]#comment]
[url.onion[:port]#comment]
[...]
"
"url.onion[:port]#comment"
[...]
Some explanation. Same in other words…
-
Most lines are single lines such as: “url.onion[:port]#comment”
– I.e. onion url optionally followed by a :port, for example some.onion:443
– Everything behind the # is the comment field.
-
What are the comments good for? For example 33y6fjyhs3phzfjj.onion is hosted by The Guardian. Anyone auditing the sdwdate log would wonder what 33y6fjyhs3phzfjj.onion is, who hosts that. If there was no comment, it would be super tedious to figure that out. The comment field also contains where the information about the existence of that .onion domain come from. Somewhat “proof of inheritance”. Full example: “33y6fjyhs3phzfjj.onion#The Guardian https://securedrop.theguardian.com 33y6fjyhs3phzfjj.onion”
-
Sometimes multiple .onion domains are hosted by the same entity/individual. In that case, those are grouped within quotes. Full example.
"
atlas777hhh7mcs7.onion:80#Hosted by Thomas White. [Whonix-devel] Hidden Service as Time Source
compass6vpxj32p3.onion:80#Hosted by Thomas White. [Whonix-devel] Hidden Service as Time Source
globe223ezvh6bps.onion:80#Hosted by Thomas White. [Whonix-devel] Hidden Service as Time Source
bbbbbb6qtmqg65g6.onion:80#Hosted by Thomas White. [Whonix-devel] Hidden Service as Time Source
pppppptkftqqnfsq.onion:80#Hosted by Thomas White. [Whonix-devel] Hidden Service as Time Source
"
sdwdate sees them as “one”. Not as many. Should that entity be picked, a random one (in the above example number x of 5 total) is picked from that group.
We can change the format if that makes parsing easier, but I would like to keep the features.
- write / test the parsing function in url_to_unixtime.
Let’s keep parsing out of url_to_unixtime, let sdwdate do that? url_to_unixtime could stay as simple as is in current version of sdwdate. sdwdate would just call url_to_unixtime as it currently does.
Otherwise if we make url_to_unixtime a module, not forking system calls, the module also should stay simple so it can be re-used elsewhere (sdwdate, unit test).
- move it in a separate module, because it should be used by both sdwdate and url_to_unixtime in standalone mode (check the sites in the pools).
sdwdate
url_to_unixtime
unit_test
sdwdate and unit_test would both be users of the url_to_unixtime module.
Config parsing, pools parsing could be separate modules.
[quote=“Patrick, post:9, topic:1137”][quote author=Patrick link=topic=1301.msg8494#msg8494 date=1434294186]
- drop built-in (hardcoded) pools, just refuse to work if config file does not exist?
[/quote]
In other words…
Drop:
Keep:
In its threaded version, url_to_unixtime expects a url list in a single parameter and returns unixtimes one by one in random order.
[That should be modified so that it returns only a list with valid times.]
In bash, sdwdate should build the url list (in the loop), pass it to url_to_unixtime, wait for the unixtime list and perform the existing checks and calculations.
In python, the list can be passed as a single argument (list type), but I guess that in bash, it’s possible to create a comma separated list of strings, that could be parsed in url_to_unixtime.
The explanation about the config file makes it clearer. Will try to keep it d-style.
Let's keep parsing out of url_to_unixtime, let sdwdate do that? url_to_unixtime could stay as simple as is in current version of sdwdate. sdwdate would just call url_to_unixtime as it currently does.
In the final version, url_to_unixtime should be stripped of any system call, stay as simple as the current version.
Asynchronous is really worth doing, now that with systemd the boot times are so fast, time sync would be the major time sunk.
So we want to keep bash in the first iteration or kick it out from start? I assume we make it python-only from start for my following words. If we do it python-only and without forking system calls, then url_to_unixtime does no longer need a command line interface either.
It’s 3 threads and 3 pools. url_to_unixtime should really be just be url to unixtime. Really just fetch 1 unixtime from 1 onion domain. Not 3_url_to_3_unixtime. There could be a 3_url_to_3_unixtime module that uses url_to_unixtime. 3_url_to_3_unixtime would do the threading and calling of url_to_unixtime. No need to call it 3_url_to_3_unixtime, though, it could just be done in sdwdate.
sdwdate_loop currently picks a random entity from a pool. $random_x from $total_pool_members. Asks sdwdate_get_time_from_remote to try fetching the time. If it works… Fine. Done. Otherwise the entity is remembered for that run and ignored. Another $random_x from $total_pool_members gets chosen. (Should it be an already remembered one, another attempt to pick a random one will be made.)
If too many pool members were unreachable, the pool is considered broken and sdwdate aborts. That means, time will not be set. Will be logged. Then sdwdate returns back to the waiting loop and tries again when the random sleep period (see sdwdate_sleep).
Getting that logic done right is non-trivial, but doable. Test cases need to involve no pool members, many pool members, various values for ALLOWED_PER_POOL_FAILURE_RATIO.
When doing asynchronous requests, it could happen that 1, 2 or 3 pools are unreachlable. Either pool one, two and/or three could be down. As soon this is recognized, should all fetches be aborted or should sdwdate wait for all three threads to return?
off-topic: mailed you a while ago.
So we want to keep bash in the first iteration or kick it out from start? I assume we make it python-only from start for my following words.
Yes, python only would be easier.
If we do it python-only and without forking system calls, then url_to_unixtime does no longer need a command line interface either.
Yes, see below.
I believe that we could make sdwdate logic simpler, with the help of python-gevent.
There are two files in GitHub - troubadoour/url_to_unixtime-asynchronous. You should create them in the same directory, for the import.
- the structure of url_to_unixtime should be more or less final.
- in test.py, we just pass the url list from the pool to url_to_unixtime and wait for the returned list of valid unixtimes.
python-gevent opens the sockets concurrently, and the the fetching order is random (the sockets are not open as ordered in the list).
So with this figure, sdwdate_loop is not required any longer, as well as the randomization. which is done by gevent. The “remember bad url” does not look necessary at that point, as gevent uses some sort of brute force, and the acquisition time penalty is minimal, providing that not too many remotes are down. We could probably create a log for the bad sites.
Then, from the returned list, we could start building up the sdwate remaining checks and functions.
A question: what is the reason to separate the remotes in three pools? From what I understand from sdwdate code, all the remotes are fetched.
Note. commenting lines 57-58 in test.py let us display only the failing remotes. When run successively several times, some of them are consistently down, with a different error, though.
Note, the ‘remembering’ is only in place during an iteration. Not during the whole lifetime of the daemon. (That would be bad in case of network issues, all would be remembered as down, even though they are not. No more time sync therefore.)
python-gevent opens the sockets concurrently, and the the fetching order is random (the sockets are not open as ordered in the list).
So with this figure, sdwdate_loop is not required any longer, as well as the randomization. which is done by gevent. The “remember bad url” does not look necessary at that point, as gevent uses some sort of brute force, and the acquisition time penalty is minimal, providing that not too many remotes are down. We could probably create a log for the bad sites.
But you’re not planing to try to connect to all entities of one pool at once? That would cause lots of extra traffic for all servers and clients if you think this for whole Whonix distribution users.
A question: what is the reason to separate the remotes in three pools? From what I understand from sdwdate code, all the remotes are fetched.
1 member from each pool is fetched per iteration. (Unless a member is down, then others are tried.)
Pools are split for better security. Those are supposed to be run by diverse entities. Split into groups, that ideally won’t ever collaborate in an attack.
I was slightly confused. getting clearer. Back to the drawing board.
Pushed a new test script in GitHub - troubadoour/url_to_unixtime-asynchronous.
[If the logic is sound, will create a new package, or a branch in sdwdate]
Both files are modified.
“test.py” is the first attempt to a python sdwdate_loop. It fetches one unique remote from each pool. Most of the debugging print statements are left, so that you can follow the internals.
The process is fast, and with timeout implemented in url_to_unixtime, does not seem to hang (for one minute or so), but there is one small issue with gevent timeout. Looking into it.
Will test this soon.
Could you please superficially look through the open sdwdate tickets?
https://phabricator.whonix.org/maniphest/?statuses=open()&projects=PHID-PROJ-25ql66l3fo7il6prq6so#R
New plugins are planned. One day.
It’s nothing that should be done right now. Probably nothing important to consider for now. Just to avoid coming up with a design that would later make it difficult to add these stuff (holes for the plugins).
I guess it’s a good start, but needs more work with logging and etc.
Median. Not average. Average is a bug, because then one pool being very much off would still have a huge impact.
Can you keep the metadata in the pools?
A good test case if having the gateway shut down. (Or Tor shut down or else…) test.py never exits then.
Issue: duplicates are added to the remembered list.
And causes high cpu then due to the endless loop.
Created a new branch in sdwdate. GitHub - troubadoour/sdwdate at python
Waiting for their proper location, the files are in /usr/lib/sdwdate-python.
Yes, I remember Whonix Forum
Can you keep the metadata in the pools?
At this stage, the original file 30_sdwdate_default is parsed (not quite finished).
A good test case if having the gateway shut down. (Or Tor shut down or else...) test.py never exits then.
Fixed. Two additions with comments in url_to_unixtime.py (line 114) and sdwdate_python (line 142).
Issue: duplicates are added to the remembered list.
Could not see any duplicates. For me, the "pool n picked urls" contain unique members.
(As per initial commit, exit if tor is not running. · troubadoour/sdwdate@0e8fcd5 · GitHub…)
Still having endless loop issues.
For reproduction, use this test case als pool_one.
pool_one = ['a',
'b',
'c',
'd',
'e',
'f',
'g',
'h',
'i',
'k',
'l',
'm',
'n',
'o',
'p',
'q']