I’ve been wondering for a while now, if
“Bad urls” was bad. Replaced by “Unreachable” for not finding a better word myself. If someone can make a suggestion…
[quote=“Patrick, post:141, topic:1137”]I’ve been wondering for a while now, if
self.range_pool_one
self.already_picked_index_pool_one
self.pool_one_done
self.url_random_pool_one
The solution would be to declare new arrays:
self.range_pool = [] for i in range(self.number_of_pools)
self.already_picked_index_pool = [] for i in range(self.number_of_pools)
and so on.
Still left with the single variables returned from config, used in the new function. That should be feasible, with a major rewrite. Putting it in the pipeline.
[quote=“Patrick, post:141, topic:1137”]I’ve been wondering for a while now, if
The simplest thing would be to merge all that pool_one, pool_two and pool_three to a single array pools[]. And instead of using separate arrays for each attribute, you can create a class for them:
class Pool:
def __init__(self):
name = "" # 'one', 'two', 'three' - for logging purposes; or use numbers instead and drop this field
url = []
url_random = []
comment = []
done = False
already_picked_index = []
@property
def range(self):
return len(self.url)And then iterate with something like for pool in self.pools:
Also there could be some other improvements:
The first code chunk:
[ul][li]use “random.randrange(0, pool[x].range)” instead of “random.sample(range(…), 1)”[/li]
[li]I don’t see why while loop is there - since “self.already_picked_index_pool_two.append(url_index)” it will execute exactly one iteration[/li][/ul]
The second code chunk:
[ul][li]“for i in range(len(self.url_random_pool_one))” → “for url_random in pool.url_random” (since you don’t use “i” in anything other than url_random_pool_one[]) [/li][/ul]
Thanks for the input, it’s quite helpful.
sdwdate is modified. There is a liitle more involved than creating a class. “config.py” is modified too, to return the pool’s data individually. It means that we have to read the whole lot three times.
I have commited each siginficant step (11 commits).
New commits.
[quote=“troubadour, post:142, topic:1137”]“Bad urls” was bad. Replaced by “Unreachable” for not finding a better word myself. If someone can make a suggestion…
The problem is that each pool has single variables that would have to be passed individually.
self.range_pool_one
self.already_picked_index_pool_one
self.pool_one_done
self.url_random_pool_one
The solution would be to declare new arrays:
self.range_pool = [] for i in range(self.number_of_pools)
self.already_picked_index_pool = [] for i in range(self.number_of_pools)
and so on.
Still left with the single variables returned from config, used in the new function. That should be feasible, with a major rewrite. Putting it in the pipeline.[/quote]
Works well.
See this iteration.
2015-09-21 21:14:19,208 - sdwdate_log - INFO - Gradually adjusting the time by running sclockadj, PID=1021
2015-09-21 21:14:19,209 - sdwdate_log - INFO - Last run (on Mon Sep 21 21:14:19 UTC 2015) was successful. Sleeping for 10 minutes.
2015-09-21 21:24:19,471 - sdwdate_log - INFO - Fetching remote times, start Mon Sep 21 21:24:19 UTC 2015 (unixtime 1442870659.47)
2015-09-21 21:24:19,472 - sdwdate_log - INFO - The clock is sane. Current time "Mon Sep 21 21:24:19 UTC 2015".
2015-09-21 21:24:19,472 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-21 21:24:19,472 - sdwdate_log - INFO - Requested urls ['hkjpnjbvhrxjvikd.onion', '5r4bjnjug3apqdii.onion', 'wi7qkxyrdpu5cmvr.onion']
2015-09-21 21:24:25,268 - sdwdate_log - INFO - Returned urls "['hkjpnjbvhrxjvikd.onion', '5r4bjnjug3apqdii.onion', 'wi7qkxyrdpu5cmvr.onion']"
2015-09-21 21:24:25,269 - sdwdate_log - INFO - Remote status "hkjpnjbvhrxjvikd.onion", False: connect error: 0x04: Host unreachable
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Remote status "5r4bjnjug3apqdii.onion", True
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Remote status "wi7qkxyrdpu5cmvr.onion", True
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Pool 2 last url: 5r4bjnjug3apqdii.onion, web unixtime: 1442870662, web time: Mon Sep 21 21:24:22 UTC 2015, diff: -3 seconds
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Pool 3 last url: wi7qkxyrdpu5cmvr.onion, web unixtime: 1442870662, web time: Mon Sep 21 21:24:22 UTC 2015, diff: -3 seconds
2015-09-21 21:24:25,271 - sdwdate_log - INFO - Running sdwdate loop, iteration 2
2015-09-21 21:24:25,271 - sdwdate_log - INFO - Requested urls ['hkjpnjbvhrxjvikd.onion']
2015-09-21 21:24:30,100 - sdwdate_log - INFO - Returned urls "['hkjpnjbvhrxjvikd.onion']"
2015-09-21 21:24:30,101 - sdwdate_log - INFO - Remote status "hkjpnjbvhrxjvikd.onion", False: connect error: 0x04: Host unreachable
2015-09-21 21:24:30,102 - sdwdate_log - INFO - Running sdwdate loop, iteration 3
2015-09-21 21:24:30,102 - sdwdate_log - INFO - Requested urls ['pubdrop4dw6rk3aq.onion']
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Returned urls "['pubdrop4dw6rk3aq.onion']"
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Remote status "pubdrop4dw6rk3aq.onion", True
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Pool 1 last url: pubdrop4dw6rk3aq.onion, web unixtime: 1442870673, web time: Mon Sep 21 21:24:33 UTC 2015, diff: 1 seconds
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Reachable urls:
5r4bjnjug3apqdii.onion: "Irpileaks[29][30] 2013-October-7 Investigative Journalism 5r4bjnjug3apqdii.onion https://5r4bjnjug3apqdii.tor2web.org/ Italy"
wi7qkxyrdpu5cmvr.onion: "Austici www.autistici.org/en/stuff/man_anon/tor.html"
pubdrop4dw6rk3aq.onion: "ProPublica https://securedrop.propublica.org pubdrop4dw6rk3aq.onion"
2015-09-21 21:24:32,056 - sdwdate_log - INFO - Unreachable urls:
hkjpnjbvhrxjvikd.onion: "Radio24syv https://securedrop.radio24syv.dk hkjpnjbvhrxjvikd.onion"
hkjpnjbvhrxjvikd.onion: "Radio24syv https://securedrop.radio24syv.dk hkjpnjbvhrxjvikd.onion"
2015-09-21 21:24:32,056 - sdwdate_log - INFO - Fetching remote times, end Mon Sep 21 21:24:19 UTC 2015 (unixtime 1442870672.06)
2015-09-21 21:24:32,060 - sdwdate_log - INFO - Pool differences, sorted: [-3, -3, 1]
2015-09-21 21:24:32,061 - sdwdate_log - INFO - Median time difference: -3
2015-09-21 21:24:32,061 - sdwdate_log - INFO - Seconds to add: - 0.674467549
2015-09-21 21:24:32,061 - sdwdate_log - INFO - New time difference: -3.674467549
2015-09-21 21:24:32,064 - sdwdate_log - INFO - Gradually adjusting the time by running sclockadj, PID=1126
2015-09-21 21:24:32,065 - sdwdate_log - INFO - Last run (on Mon Sep 21 21:24:32 UTC 2015) was successful. Sleeping for 10 minutes.The same unreachable url was tried twice.
Please cherry-pick Make sure /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.… · Kicksecure/sdwdate@1566b7c · GitHub.
git cherry-pick 1566b7c9b7d36bdb4b48cd607c743df5d4d57fe2Another one.
Please cherry-pick use cleaner solution 'Conflicts: systemd-timesyncd' rather than 'serv… · Kicksecure/sdwdate@50fd9eb · GitHub.
git cherry-pick 50fd9eb567da5128c53d0592ee99a75334c6efa9Cherry-picked both commits.
The same unreachable url was tried twice.In the url append code block (line 305 to 321), I had removed the while loop. It prevents picking the same url twice. Reinstated. https://github.com/troubadoour/sdwdate/commit/40701d13b317be8afef300649b4351d346ba3662
As of latest commit, it’s broken. This is one restart cycle.
2015-09-24 12:24:29,166 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:29 UTC 2015 (unixtime 1443097469.17)
2015-09-24 12:24:29,171 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:29 UTC 2015".
2015-09-24 12:24:29,171 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:29,594 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:29 UTC 2015 (unixtime 1443097469.59)
2015-09-24 12:24:29,607 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:29 UTC 2015".
2015-09-24 12:24:29,608 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:30,093 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:30 UTC 2015 (unixtime 1443097470.09)
2015-09-24 12:24:30,105 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:30 UTC 2015".
2015-09-24 12:24:30,105 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:30,594 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:30 UTC 2015 (unixtime 1443097470.59)
2015-09-24 12:24:30,605 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:30 UTC 2015".
2015-09-24 12:24:30,606 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:31,094 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:31 UTC 2015 (unixtime 1443097471.09)
2015-09-24 12:24:31,105 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:31 UTC 2015".
2015-09-24 12:24:31,106 - sdwdate_log - INFO - Running sdwdate loop, iteration 1Strange. I compared the files in the remote repositiry and the one i’m using, they are equal.
I could reproduce it by commenting line 307 in sdwdate (because I forgot it at some stage beforehand).
Just confirmed, got that line. (And the issue.)
Have deleted all the sdwdate files, removed the local repositiry, cloned from troubadoour and reinstalled. Have’nt got the issue.
So, step by step. Reverted the last commit and pushed. Can you check again?
Issue persists.
After starting with a fresh VM / reboot, it works. So this could be an issue with packaging or python caching or something.
Also the previous 7dce7b7902d6b53b8c83d84cea195f3b522a5af1 seems to work now.
I mean 40701d13b317be8afef300649b4351d346ba3662.
Glad it works! Have hard reset the revert anf force pushed.
Logging pid on start.
By the way, tried using python-daemon so that the pid would be automatically managed, but it clashes with gevent in url_to _unxtime. It seems that there is no solution at this stage of development, be it for python-daemon or python-gevent.
Regarding the use of gui_message, not sure it should be implemented in sdwdate. A lot of the messages are for logging only and do not require HTML But the question is: do we want all the GUI verbose in the log, especially for situations where the user is asked to fix an issue, like in timesanitycheck failing? Perhaps gui_message can be used in sdwdate-gui only.
It’s finally implemented in sdwdate, because of the redundancies it would create with the log messages.
new file sdwdate.yaml · troubadoour/sdwdate@f9a735a · GitHub and
use external yaml file for logging / sdwdate-gui messages · troubadoour/sdwdate@75f20cb · GitHub
If there is anything you wish to add…
Regarding timesanitycheck, will look in a gui for manually set the time, and run it after fetching the times.