sdwdate and sdwdate-gui development thread

Pushed a batch of commits.

  • /etc/sdwdate.d instead of /etc/sdwdate-python.d
  • cleanup etc/sdwdate.d/30_sdwdate_default.conf
  • prerequisite_check works with te_pe_tb_check (thanks :slight_smile: )

The error messages from te_pe_tb_check are passed to sdwdate-gui as is, without formatting. It seems that the script is used by sdwdate only. As far as I understand, whonixcheck calls tor_bootstrap_check.bsh and tor_enabled_check directly. The question is: can we modify the messages in te_pe_tb_check with some HTML, for better output in sdwdate-gui?

can we modify the messages in te_pe_tb_check with some HTML, for better output in sdwdate-gui?

Yes.

[hr]

What about…

etc/sdwdate.d/30_sdwdate_default.conf

etc/sdwdate.d/30_default.conf

? (To not re-do a perhaps earlier bug. Why should sdwdate be twice in the absolute filename?)

[hr]

This comment we should re-add.

[code]## Please use “/etc/sdwdate.d/50_user.conf” for your custom

configuration, which will override the defaults found here.

When sdwdate is updated, this file may be overwritten.[/code]

Currently the log gets spammed with the same message. Example:

sdwdate-bash used “safelog”, i.e. not re-add the very same message again for some messages such as the ones from te_pe_tb_check.

A few commits:

  • add HTML in te_pe_tb_check te_pe_tb_check: HTML in messages · troubadoour/anon-shared-helper-scripts@c17e562 · GitHub
  • modify prerequisite_check: accordingly. The message in the log is shown with new lines like in sdwdate-gui, because of the “
    ” “\n” substitution in strip_html. That could be removed, but in case of timesanitycheck failure, the whole (long) message would be on a single line, not very readable.
  • renam /etc/sdwdate.d/30_sdwdate_default.conf to /etc/sdwdate.d/30_default.conf.
  • re-add the comment for custom .d files.
  • one shot logging in prerequisite_check (safelog)

[quote=“Patrick, post:171, topic:1137”]Cannot use port 9108 for everyone by default. Won’t work on plain Debian. Only 9050 is available there by default. (And editing torrc is not allowed for packages.)

IP / port should be overwriteable when using it config. Currently:
PROXY_IP
PROXY_PORT
(We can consider to comment those out by default for making it easier to have the built in defaults.)[/quote]
Wondering how to implement that. In an external file? In /etc/sdwdate.d?

Had to change the ssh key in github because of this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Please contact your system administrator. Add correct host key in /home/USER/.ssh/known_hosts to get rid of this message. Offending key in /home/USER/.ssh/known_hosts:1 RSA host key for stash.customer.com has changed and you have requested strict checking. Host key verification failed. fatal: The remote end hung up unexpectedly
Made a quick search, not really conclusive. Have you ever experienced that?

Wondering how to implement that. In an external file? In /etc/sdwdate.d?
Just in the normal config file.

Perhaps out commented by default. Then if PROXY_IP or PROXY_PORT is set in the config, just use these values, no auto detection.

For auto detection:

  • Running in Whonix (i.e. if folder /usr/share/whonix exists) → port → 9108.
  • Otherwise, running outside Whonix → port → 9050.

About Whonix Forum :

  • yes, I saw this in my life before either because I migrated ssh user data from one system to another and messed up something or because remote changed the fingerprint
  • that happens if remote (here: github) changed their ssh key. But they did not. At least not for the servers I am reaching. (They could be using ssh round robin, different data centers with different ssh keys.)
  • you creating a new key should not be necessary
  • just verify that the ssh fingerprint you are receiving is the one that you are expecting
    – here: Connecting to GitHub with SSH - GitHub Docs - relevant part: "
    The authenticity of host ‘github.com (207.97.227.239)’ can’t be established.

RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.

"

  • could really be a man in the middle attack. In that case nothing too bad happened except for the denial of service. In that case no changes on your system required. Just an outgoing connection that does not tamper with connections. Perhaps just another Tor exit?

Added proxy settings (commented) in /etc/sdwdate.d/30_default.conf.

A new module proxy_settings.py returns the settings (from conf or auto detected) to sdwdate.

sdwdate_loop pass the settings to remote_times.

In check for general errors only on first fetching iteration (three urls… · troubadoour/sdwdate@81e9e6b · GitHub, general_proxy_error and general_timeout_error are checked on the first iteration only (or if three urls are requested). There was a bug popping a general_timeout_error once in a while, and it’s more logical to do it that way. Modified [no_valid_time returned from pool] message for the case tor or internet goes down during an acquisition cycle.

Looks good so far, but the port auto detection needs more work.

if port_number == '': port_number = '9108'

That should default to 9050 to support non-Whonix distributions where Tor by default lists only on 9050. And above the fallback code, if Whonix is detected (i.e. if folder /usr/share/whonix exists) -> port -> 9108.

[hr]

  • prerequisite_check message should start with “prerequisite_check:” or something like this so we can identify, that messages are coming form that check.

  • I think temp_dir needs to be deleted afterwards. Otherwise we create a million of those. It would be best to create it just once globally during startup and delete it on exit.

  • Do we want to support with/without having anon-shared-helper-scripts installed?
    – If no -> needs “Depends: anon-shared-helper-scripts”
    – If yes -> Skip if that file is not executable.
    – Support outside of Whonix would be difficult, since setting up ControlPort access would have to be set up, which is difficult in the absence of the tor.d config folder feature.

[quote=“Patrick, post:189, topic:1137”]Looks good so far, but the port auto detection needs more work.

if port_number == '': port_number = '9108'

That should default to 9050 to support non-Whonix distributions where Tor by default lists only on 9050. And above the fallback code, if Whonix is detected (i.e. if folder /usr/share/whonix exists) → port → 9108.[/quote]
The script follows the same logic for ip and port settings, that is:

  • if PROXY_IP commented in configuration (ip_address = ’ ') → autodetect ip, else use PROXY_IP value.
  • if PROXY_PORT commented in configuration (port_number = ’ ') → use Whonix default 9108, else use PROXY_PORT value.

We should revert the logic to make it default to 9050.

- prerequisite_check message should start with "prerequisite_check:" or something like this so we can identify, that messages are coming form that check.
https://github.com/troubadoour/anon-shared-helper-scripts/commit/9428ed0a3ec436d4dd49c7728d9b246d2f48e589
- I think temp_dir needs to be deleted afterwards. Otherwise we create a million of those. It would be best to create it just once globally during startup and delete it on exit.
  • Do we want to support with/without having anon-shared-helper-scripts installed?
    – If no → needs “Depends: anon-shared-helper-scripts”
    – If yes → Skip if that file is not executable.
    – Support outside of Whonix would be difficult, since setting up ControlPort access would have to be set up, which is difficult in the absence of the tor.d config folder feature.

temp_dir created on start and removed in sigterm handler.
prereqisite_check run only if “/usr/lib/anon-shared-helper-scripts/te_pe_tb_check” exists (instead of [if file executable]).
class PreCheckVariables: set run_prerequite true and create temp_dir … · troubadoour/sdwdate@9f3a4a6 · GitHub
We should revert the logic to make it default to 9050.
Yes.
https://github.com/troubadoour/anon-shared-helper-scripts/commit/9428ed0a3ec436d4dd49c7728d9b246d2f48e589
Why not implement this at the python level? In conceptual term, the idea is to call te_pe_tb_check the prereqisite_check. Sdwdate obtains the results from prereqisite_check. Then to mark that sdwdate did not obtain the messages itself, these are prefixed with "Prerequisite check:". But prefixing should be up to sdwdate. Because te_pe_tb_check could return no output or strange output (errors) which it would not prefix itself. Also then the prereqisite_check is more configurable, modular. Only the path to the script would have to be replaced, nothing else.
prereqisite_check run only if "/usr/lib/anon-shared-helper-scripts/te_pe_tb_check" exists (instead of [if file executable]).
Can we make this if executable? The idea is being able to easily disable the check for testing purposes.

usr/share/test_d_files is outdated.

ALLOWED_PER_POOL_FAILURE_RATIO is not implemented yet?

Yes.[/quote]
I have a problem with that. If we say

if "/usr/share/whonix" exists: port = 9108 else: port = 9050
that makes reading the proxy settings in the configuration file redundant.

We could apply the same to the ip.

if "/usr/share/whonix" exists: autodetect ip else: ip = 127.0.0.1

Can do. The ability to set an arbitrary IP / port in the config is not so important, mostly useful for use cases outside of Whonix.

Done, with three options: whonix, configuration file proxy settings uncommented, default to [127.0.01:9050].

The “Prerequisite check” messages header was pushed in the same commit.

Removed “Prerequisite check” messages header form te_pe_tb_check.

Run prerequisite_check if te_pe_tb_check is executable.

ALLOWED_PER_POOL_FAILURE_RATIO is not implemented.

Can we safely rename “te_pe_tb_check” to “prerequisite_check”, or “sdwdate_prerequisite_check”?

Missing dot.

ip_address = '1270.0.1'
Can we safely rename "te_pe_tb_check" to "prerequisite_check", or "sdwdate_prerequisite_check"?
"sdwdate" should not be included in the name, because it's shipped by a generic package anon-shared-helper-scripts.

It can be renamed when we either rename it in sdwdate-bash in meanwhile or once sdwdate-python replaces sdwdate-bash.

Pushed three commits:

As far as I can see, next could be a test unit for devs, to check all the servers, and I still have to try custom .d configuration files
Plus, on top from what you might want to change or add, there is some cleaning and pep8 checks before it can be prereleased.

Works good.

[hr]

Sorry, but I think I messed something up. The Tor/gateway IP detection. It actually got quite complex. Non-Qubes-Whonix vs Qubes-Whonix + Qubes gateway vs Qubes workstation + Qubes gateway-template + Qubes workstation-template vs no net-vm configured.

[[(Also whonixcheck has to deal with all these cases, see https://github.com/Whonix/whonixcheck/blob/master/usr/lib/whonixcheck/preparation and look for ‘GATEWAY_IP=’. As well as tb-updater has to deal with this.)]]

The whole code in bash is this:

And this code may change in future. Then we would have to re-implement this in python as well as in bash. So I would like to refactor this.

I’ve created /usr/lib/anon-shared-helper-scripts/settings_echo. (https://github.com/Whonix/anon-shared-helper-scripts/blob/master/usr/lib/anon-shared-helper-scripts/settings_echo)

Example run:

/usr/lib/anon-shared-helper-scripts/settings_echo
TOR_CONTROL_HOST="10.137.2.1"
TOR_CONTROL_AUTHENTICATE="0"
TOR_CONTROL_PORT="9052"

This is how it could be interfaces from bash:

Set.

eval $(/usr/lib/anon-shared-helper-scripts/settings_echo)

Echo.

echo "$TOR_CONTROL_HOST"

Result.

10.137.2.1

So maybe in sdwdate-python we could implement it as this:

  • if folder /usr/share/whonix exists + if file /usr/lib/anon-shared-helper-scripts/settings_echo is executable + then run /usr/lib/anon-shared-helper-scripts/settings_echo and set ip_address = $TOR_CONTROL_HOST.
  • otherwise ip_address = PROXY_IP
  • fall back to 127.0.0.1

What do you think?

Makes the python code easier to read. :slight_smile:

Something strange. When calling “/usr/lib/anon-shared-helper-scripts/settings_echo” from a terminal in Workstation, it returns

TOR_CONTROL_HOST="127.0.0.1".
When called from the python script, we get the correct value “10.152.152.10”.

In Gateway, it returns “127.0.0.1”, as expected.

Your gpg key expired. Please renew.