[HOME] [DOWNLOAD] [DOCS] [NEWS] [SUPPORT] [TIPS] [ISSUES] [Priority Support]

sdwdate and sdwdate-gui development thread


#141

I’ve been wondering for a while now, if


#142

“Bad urls” was bad. Replaced by “Unreachable” for not finding a better word myself. If someone can make a suggestion…

[quote=“Patrick, post:141, topic:1137”]I’ve been wondering for a while now, if

self.range_pool_one self.already_picked_index_pool_one self.pool_one_done self.url_random_pool_one

The solution would be to declare new arrays:

self.range_pool = [] for i in range(self.number_of_pools) self.already_picked_index_pool = [] for i in range(self.number_of_pools)
and so on.

Still left with the single variables returned from config, used in the new function. That should be feasible, with a major rewrite. Putting it in the pipeline.


#143

[quote=“Patrick, post:141, topic:1137”]I’ve been wondering for a while now, if

The simplest thing would be to merge all that pool_one, pool_two and pool_three to a single array pools[]. And instead of using separate arrays for each attribute, you can create a class for them:

class Pool:
  def __init__(self):
    name = "" # 'one', 'two', 'three' - for logging purposes; or use numbers instead and drop this field
    url = []
    url_random = []
    comment = []
    done = False
    already_picked_index = []
    
  @property
  def range(self):
    return len(self.url)

And then iterate with something like for pool in self.pools:

Also there could be some other improvements:
The first code chunk:

[ul][li]use “random.randrange(0, pool[x].range)” instead of “random.sample(range(…), 1)”[/li]
[li]I don’t see why while loop is there - since “self.already_picked_index_pool_two.append(url_index)” it will execute exactly one iteration[/li][/ul]

The second code chunk:

[ul][li]“for i in range(len(self.url_random_pool_one))” -> “for url_random in pool.url_random” (since you don’t use “i” in anything other than url_random_pool_one[]) [/li][/ul]


#144

Thanks for the input, it’s quite helpful.

sdwdate is modified. There is a liitle more involved than creating a class. “config.py” is modified too, to return the pool’s data individually. It means that we have to read the whole lot three times.

I have commited each siginficant step (11 commits).


#145

New commits.

  • logging pool number in “no valid time returned from pool”
  • modified general_proxy_error handling.
  • added general_timeout_error
  • removed retrying loop.

#146

[quote=“troubadour, post:142, topic:1137”]“Bad urls” was bad. Replaced by “Unreachable” for not finding a better word myself. If someone can make a suggestion…

The problem is that each pool has single variables that would have to be passed individually.

self.range_pool_one self.already_picked_index_pool_one self.pool_one_done self.url_random_pool_one

The solution would be to declare new arrays:

self.range_pool = [] for i in range(self.number_of_pools) self.already_picked_index_pool = [] for i in range(self.number_of_pools)
and so on.

Still left with the single variables returned from config, used in the new function. That should be feasible, with a major rewrite. Putting it in the pipeline.[/quote]
Works well.


#147

See this iteration.

2015-09-21 21:14:19,208 - sdwdate_log - INFO - Gradually adjusting the time by running sclockadj, PID=1021
2015-09-21 21:14:19,209 - sdwdate_log - INFO - Last run (on Mon Sep 21 21:14:19 UTC 2015) was successful. Sleeping for 10 minutes.
2015-09-21 21:24:19,471 - sdwdate_log - INFO - Fetching remote times, start Mon Sep 21 21:24:19 UTC 2015 (unixtime 1442870659.47)
2015-09-21 21:24:19,472 - sdwdate_log - INFO - The clock is sane. Current time "Mon Sep 21 21:24:19 UTC 2015".
2015-09-21 21:24:19,472 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-21 21:24:19,472 - sdwdate_log - INFO - Requested urls ['hkjpnjbvhrxjvikd.onion', '5r4bjnjug3apqdii.onion', 'wi7qkxyrdpu5cmvr.onion']
2015-09-21 21:24:25,268 - sdwdate_log - INFO - Returned urls "['hkjpnjbvhrxjvikd.onion', '5r4bjnjug3apqdii.onion', 'wi7qkxyrdpu5cmvr.onion']"
2015-09-21 21:24:25,269 - sdwdate_log - INFO - Remote status "hkjpnjbvhrxjvikd.onion", False: connect error: 0x04: Host unreachable
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Remote status "5r4bjnjug3apqdii.onion", True
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Remote status "wi7qkxyrdpu5cmvr.onion", True
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Pool 2 last url: 5r4bjnjug3apqdii.onion, web unixtime: 1442870662, web time: Mon Sep 21 21:24:22 UTC 2015, diff: -3 seconds
2015-09-21 21:24:25,270 - sdwdate_log - INFO - Pool 3 last url: wi7qkxyrdpu5cmvr.onion, web unixtime: 1442870662, web time: Mon Sep 21 21:24:22 UTC 2015, diff: -3 seconds
2015-09-21 21:24:25,271 - sdwdate_log - INFO - Running sdwdate loop, iteration 2
2015-09-21 21:24:25,271 - sdwdate_log - INFO - Requested urls ['hkjpnjbvhrxjvikd.onion']
2015-09-21 21:24:30,100 - sdwdate_log - INFO - Returned urls "['hkjpnjbvhrxjvikd.onion']"
2015-09-21 21:24:30,101 - sdwdate_log - INFO - Remote status "hkjpnjbvhrxjvikd.onion", False: connect error: 0x04: Host unreachable
2015-09-21 21:24:30,102 - sdwdate_log - INFO - Running sdwdate loop, iteration 3
2015-09-21 21:24:30,102 - sdwdate_log - INFO - Requested urls ['pubdrop4dw6rk3aq.onion']
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Returned urls "['pubdrop4dw6rk3aq.onion']"
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Remote status "pubdrop4dw6rk3aq.onion", True
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Pool 1 last url: pubdrop4dw6rk3aq.onion, web unixtime: 1442870673, web time: Mon Sep 21 21:24:33 UTC 2015, diff: 1 seconds
2015-09-21 21:24:32,055 - sdwdate_log - INFO - Reachable urls:
5r4bjnjug3apqdii.onion: "Irpileaks[29][30]      2013-October-7  Investigative Journalism        5r4bjnjug3apqdii.onion  https://5r4bjnjug3apqdii.tor2web.org/   Italy"
wi7qkxyrdpu5cmvr.onion: "Austici www.autistici.org/en/stuff/man_anon/tor.html"
pubdrop4dw6rk3aq.onion: "ProPublica     https://securedrop.propublica.org       pubdrop4dw6rk3aq.onion"
2015-09-21 21:24:32,056 - sdwdate_log - INFO - Unreachable urls:
hkjpnjbvhrxjvikd.onion: "Radio24syv     https://securedrop.radio24syv.dk        hkjpnjbvhrxjvikd.onion"
hkjpnjbvhrxjvikd.onion: "Radio24syv     https://securedrop.radio24syv.dk        hkjpnjbvhrxjvikd.onion"
2015-09-21 21:24:32,056 - sdwdate_log - INFO - Fetching remote times, end Mon Sep 21 21:24:19 UTC 2015 (unixtime 1442870672.06)
2015-09-21 21:24:32,060 - sdwdate_log - INFO - Pool differences, sorted: [-3, -3, 1]
2015-09-21 21:24:32,061 - sdwdate_log - INFO - Median time difference: -3
2015-09-21 21:24:32,061 - sdwdate_log - INFO - Seconds to add: - 0.674467549
2015-09-21 21:24:32,061 - sdwdate_log - INFO - New time difference: -3.674467549
2015-09-21 21:24:32,064 - sdwdate_log - INFO - Gradually adjusting the time by running sclockadj, PID=1126
2015-09-21 21:24:32,065 - sdwdate_log - INFO - Last run (on Mon Sep 21 21:24:32 UTC 2015) was successful. Sleeping for 10 minutes.

The same unreachable url was tried twice.


#148

Please cherry-pick https://github.com/Whonix/sdwdate/commit/1566b7c9b7d36bdb4b48cd607c743df5d4d57fe2.


#149

Another one.
Please cherry-pick https://github.com/Whonix/sdwdate/commit/50fd9eb567da5128c53d0592ee99a75334c6efa9.


#150

Cherry-picked both commits.

The same unreachable url was tried twice.
In the url append code block (line 305 to 321), I had removed the while loop. It prevents picking the same url twice. Reinstated. https://github.com/troubadoour/sdwdate/commit/40701d13b317be8afef300649b4351d346ba3662

#151

As of latest commit, it’s broken. This is one restart cycle.

2015-09-24 12:24:29,166 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:29 UTC 2015 (unixtime 1443097469.17)
2015-09-24 12:24:29,171 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:29 UTC 2015".
2015-09-24 12:24:29,171 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:29,594 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:29 UTC 2015 (unixtime 1443097469.59)
2015-09-24 12:24:29,607 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:29 UTC 2015".
2015-09-24 12:24:29,608 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:30,093 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:30 UTC 2015 (unixtime 1443097470.09)
2015-09-24 12:24:30,105 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:30 UTC 2015".
2015-09-24 12:24:30,105 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:30,594 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:30 UTC 2015 (unixtime 1443097470.59)
2015-09-24 12:24:30,605 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:30 UTC 2015".
2015-09-24 12:24:30,606 - sdwdate_log - INFO - Running sdwdate loop, iteration 1
2015-09-24 12:24:31,094 - sdwdate_log - INFO - Fetching remote times, start Thu Sep 24 12:24:31 UTC 2015 (unixtime 1443097471.09)
2015-09-24 12:24:31,105 - sdwdate_log - INFO - The clock is sane. Current time "Thu Sep 24 12:24:31 UTC 2015".
2015-09-24 12:24:31,106 - sdwdate_log - INFO - Running sdwdate loop, iteration 1

#152

Strange. I compared the files in the remote repositiry and the one i’m using, they are equal.

I could reproduce it by commenting line 307 in sdwdate (because I forgot it at some stage beforehand).


#153

Just confirmed, got that line. (And the issue.)


#154

Have deleted all the sdwdate files, removed the local repositiry, cloned from troubadoour and reinstalled. Have’nt got the issue.

So, step by step. Reverted the last commit and pushed. Can you check again?


#155

Issue persists.


#156

After starting with a fresh VM / reboot, it works. So this could be an issue with packaging or python caching or something.


#157

Also the previous 7dce7b7902d6b53b8c83d84cea195f3b522a5af1 seems to work now.


#158

I mean 40701d13b317be8afef300649b4351d346ba3662.


#159

Glad it works! Have hard reset the revert anf force pushed.

Logging pid on start.

By the way, tried using python-daemon so that the pid would be automatically managed, but it clashes with gevent in url_to _unxtime. It seems that there is no solution at this stage of development, be it for python-daemon or python-gevent.

Regarding the use of gui_message, not sure it should be implemented in sdwdate. A lot of the messages are for logging only and do not require HTML But the question is: do we want all the GUI verbose in the log, especially for situations where the user is asked to fix an issue, like in timesanitycheck failing? Perhaps gui_message can be used in sdwdate-gui only.


#160

It’s finally implemented in sdwdate, because of the redundancies it would create with the log messages.
https://github.com/troubadoour/sdwdate/commit/f9a735a26d2a872a105e7d599d1a89e5f55d0e62 and
https://github.com/troubadoour/sdwdate/commit/75f20cb991344f5ba86abc0dd756d8d70be43002

If there is anything you wish to add…

Regarding timesanitycheck, will look in a gui for manually set the time, and run it after fetching the times.