Locking down your SSH Server and Client

What options are their for locking down your Whonix ssh client.

Yubikey: Not recommended since it could leak key material. Maybe 2fa without use of ssh key?

https://whonix.org/wiki/Dev/yubikey#YubiKey_Info

Yubikeys supports storing OpenPGP (GnuPG / gpg) keys, OpenSSH keys, but I wouldn’t trust it. Another yubikey model had a PIN bypass bug. ( SecurityAdvisory 2015-04-14 )

Qubes split-ssh: Hasn’t been maintained for 2+ years. Has multiple forks but those people are unknown (to me).

ip/nf-tables: on both firewall and client box. Maybe designate an interface on firewall for ssh only. Block everything else.

Small tweaks: Change ssh to different port, strong password for key access etc…

Anythig else??

1 Like

For reference:

The value of 2FA is imo non-zero, low. Tried to flash that out for when it is useful and when not useful here: Two-factor Authentication (2FA)

I speculate if you’re creating such a threat here, you might be in the same group of users who don’t fall for attacks that would be protected by 2FA.

Either that or any other kind of 2FA. Conindently today I was looking into all pam modules available from packages.debian.org. In result, I created a list of promising candidates for use with 2FA. While I didn’t look closely at any of these at first sight these seem to use open standards, sane, libre, you name it.

Two-factor Authentication (2FA)

Nothing too special about yubikey here. Even a (separate) (dedicated) (old) phone(s) with TTOP (commonly known as “google authenticator” / AndOTP) could very likely be used for Linux PAM, which looks great. Basically any form of authentication modules can be stacked in any configuration for any way of authentication (virtual terminal (login), xscreensaver, sudo, su, lightdm, ssh). I.e. either

  • password only,
  • password or 2FA,
  • password and 2FA
  • password + TTOP + fingerprint

What works locally for login, sudo and/or su should also work (locally for testing and) on remote for ssh.

I see limited, non-zero value in split-gpg / split-ssh. It does not add much to security hardening rather limited damage control. A compromised VM cannot steal the key but use the key. In case of ssh, malware could take over an ssh session, install a backdoor on the server.
This would make more sense if that SSH key is used for different purposes such as from different VMs for different servers which are not all known at the same time by the same adversary.

1 Like

I recommend using v3 onions (keeping the addresses secret to prevent unauthorized probing of SSH server). Also connecting to it from trsted VMs. This would take care of the need to protect the token from malicious access while being straightforward.

2 Likes

page Onion Services Guides - Whonix

SSH

Secure Shell is the dominant protocol for secure remote login and system administration. It is a critical component of server and internet infrastructure.

OK.

Revelations from the Snowden documents and further analysis [1] has uncovered weaknesses in some of the included cipher-suites, allowing abuses by resourceful nation-state adversaries.

Ok interesting, but was this fixed upstream in SSH and/or does Debian nowadays ship in Debian buster secure defaults? Anything the user should do?

Do you know any secure / hardened SSH config example?

Please help improve Anonymous SSH over Tor

Here are some good SSH defaults:

/etc/ssh/ssh_config: (example with OpenSSH 7.9 Debian Buster)
Host *
ForwardAgent yes (depends on local environment circumstances)
AddressFamily inet
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
Port 22
Protocol 2
ForwardX11 no
PubkeyAuthentication yes

StrictHostKeyChecking ask
VisualHostKey yes
HashKnownHosts yes
User user
Host host
SendEnv LANG LC_*

Strongest ciphers:

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr

Most secure MACs:

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-512,hmac-sha2-256-etm@openssh.com

Secure Kex algos:

KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384
Tunnel no
#TunnelDevice any:any
#PermitLocalCommand no
#ProxyCommand ssh -q -W %h:%p gateway.example.com
End of file

Now the server part /etc/ssh/sshd_config:
Protocol 2 (Protocol 1 is dangerously outdated!)
Port 22 (22 is standard, you can make it anything. On commandline, when connecting as client, use the “-p” option if not port 22)
AddressFamily inet (means only ipv4, no v6 allowed)
ListenAddress 0.0.0.0 → this is okay because of strict firewall/ AppArmor. For an onion-only server, you listen on 127.0.0.1
#ListenAddress :: (comment out if ipv6 is disabled via sysctl or iptables or both)
HostKey /etc/ssh/ssh_host_ed25519_key

Strongest ciphers:

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr

Most secure MACs:

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-512,hmac-sha2-256-etm@openssh.com

Secure Kex algos:

KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384

Logging (Optional- okay to leave commented out)

#SyslogFacility AUTH
#LogLevel INFO (unless you need logging, leave commented out)
LoginGraceTime 2m (how long does the prompt stay if you have a passphrase to enter)
PermitRootLogin prohibit-password | no (safer to deny root obviously)
PasswordAuthentication no (use public keys ed25519 so no password authentication. A passphrase is optional for the key, but that is not part of this option. This option refers to old style password-only authentication (not secure))

StrictModes yes
MaxAuthTries 3 (number of allowed login attempts, if there is a passphrase associated with the key)
MaxSessions 4 (number of allowed sessions)
PubkeyAuthentication yes (always use public keys, not passwords)
UsePAM yes
#AllowAgentForwarding yes (circumstance specific. This refers to the ssh-agent. Useful when running a local instance and client/server exist on same machine)
#AllowTcpForwarding (be careful with this. Avoid forwarding if you can. If you do not forward tcp, leave this commented out)
TCPKeepAlive yes (dependent on connection and network. Sometimes useful, sometimes not. An okay default is to say “no” and if it turns out you need it, then do “yes.”)
X11Forwarding no
PrintMotd no
#PrintLastLog yes
#UseLogin no
#PermitUserEnvironment no
ClientAliveInterval (second amount to keep connection alive)
UseDNS no
AllowUsers user
#Banner /etc/issue.net
AcceptEnv LANG LC_*
#Subsystem sftp /usr/lib/openssh/sftp-server
GatewayPorts no
#PermitTunnel no
#ChrootDirectory none
End of File

Some points:
There are more entries that you could add depending on your needs. The one above are the most important and will establish a functioning, and well guarded setup. Use ed25519 keys only. rsa, and ecdsa are outdated. Use the ssh-keygen command to generate the keys
ssh-keygen -o -a -t ed25519
Remember like this: OATmeal is good for Ed[25519]
the -o makes the key support openssh’s own keytype not .pem
the -a determines the rounds of kdf. More=better but don’t go crazy. Generally 75-100 is good. The more rounds, the longer authentication and login will take to finish.
the -t is “type of key”
You will be asked if ~/.ssh is okay for storing the keys. Say yes.
On permissions, sometimes it gets set on its own. If not, do this:
sudo chmod 0600 ~/.ssh/id_ed25519 && sudo chmod 0644 ~/.ssh/id_ed25519.pub
Set the permission for the whole ~/.ssh folder to 0700
Create a file in ~/.ssh called authorized_keys and another one called known_hosts. Set permissions for both to 0644.
Sometimes you need to manually add your newly created private key to the ssh-agent.
ssh-add ~/.ssh/id_ed25519
The VisualHostKey yes entry in the ssh_config file makes it so when you login, openssh shows you a ascii generated picture of what the key looks like. In addition to all the other verifications, this is one last line of defense to alert you to a changed or wrong key.
Generally the newer OpenSSH’s will choose the strongest cipher, but the config files establish in what order they are parsed. Both sides need to have the same or compatible openssh versions.
Make specific firewall rules for your ssh activity too. Let’s say that you want to connect to a server at 123.45.678 and both your configs specify port 4675:
sudo iptables -A OUTPUT -p tcp -o [interface-name] -s [your client ssh ip] -d [ssh server ip] --dport 4675 --j ACCEPT
then to let them respond:
sudo iptables -A INPUT -p tcp -i [interface-name] -s [ssh server ip] -d [your client ip] --dport 4675 --j ACCEPT
This assumes a default-deny policy for INPUT and OUTPUT. If you have a default-allow for OUTPUT, then you must add this rule as the last rule in the OUTPUT chain:
sudo iptables -A OUTPUT --j DROP
This makes it so the outgoing rules you have are the only ones that allow traffic. Iptables parses in order, so when it does not see a rule matching a packet not on your list, it checks each one in the chain until it finds a match with the last rule and drops the packet.
That should set up your ssh. Some people also recommend re-generatinf the moduli file that comes in /etc/ssh. If you use the settings above, look at the kex algorithms. We are not using any diffie hellman group exchange kex algos to exchange keys, so you do not have to. The chosen kex algo is curve25519.

This could use some forum formatting using ``` for easier readability. And source.

Do you think you could edit the wiki page Remote Administration - Whonix and make this actionable instruction for ssh client users and ssh servers? I guess commented config files in wiki would be best.

Sure I could put that together; I’ll try to make it easier to follow (readable)

1 Like

I submitted an edit, but it went underneath the footnotes section by accident. Also, it numbered the options in the config files which obviously we don’t use in the real configs.
I wanted it to put the options like:
Protocol 2
but it put

  1. Protocol 2
1 Like

The formatting looks good now.

Could you please compare Debian default /etc/ssh/ssh_config and /etc/ssh/sshd_config with wiki version?

For example

  • Debian default ForwardAgent: not set
  • upstream default ForwardAgent: no
  • Whonix wiki config ForwardAgent: yes

Should only derivative from default config if we have a good reason for it which then needs to be documented with a comment above that option.


Anyone: please also read about each option here

for upstream defaults, recommendations, warnings, etc.

@Patrick,
In Debian Buster, the packages for openssh-client and openssh-server are from OpenBsd. The config that they ship is for /etc/ssh/sshd_config (the server) and it is just a list with everything commented out. It was quite outdated and incomplete.
The ssh_config has to be put together by the user based on his/her needs. To put it together, I looked in quite a few places: github, openssh’s website, stackoverflow, and in the man pages.
If both parties are using the newest current openssh packages, then a lot of the options are not needed because the client and server will negotiate the highest security cipher, mac, and kex algorithm on their own. If, the two are not the same, that’s when a reasonable fallback is required with other options common to both.
My ForwardAgent option was set to yes instead of Debian’s default because I had the client and server on the same machine, using it as a Socks5 proxy for Tor.
Here is the default openssh sshd_config:

#	$OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile	.ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem	sftp	/usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

Server config: I made my way through all settings in the suggested server config as this seems more critical than client config.

The layout for a config file with proverbially thousands of possible configuration options needs a plan. A design. It can be optimized for with different goals in mind. As an example, one conflicting goal would be brevity vs mentioning all configuration options from sshd_config man page.

Mentioning all configuration options from sshd_config would be god for some purposes but make the config file harder to maintain as sshd / sshd_config man page evolves. I guess shipping a complete sshd config file mentioning all possibilities would be too much.

Design proposal I came up with:

  • brevity
  • Defaults not mentioned in Debian default config file: don’t mention for brevity.
  • Higher security settings such as ed25519 where not the default.
  • Comments which document the reasoning behind changes.

Commonly changed settings, usability enhancements (which worsen security) would be reasonable to mention too.

That’s alright. However, if ForwardAgent is default no and setting it to yes doesn’t increase security, then we shouldn’t enable it by default. (Unless it’s a usability argument then it could be considered in theory. Maybe not for ForwardAgent but for other settings.)

Right, however when one controls both the server and the client config (image you set up a new admin VM and a new remote server) then there’s no need to take any chances for cipher negotiation. Reason is, SSL/TLS cipher negotiation had some security issues in past. Therefore I thought for purpose of hardening, just assume/demand stable version of Debian adn pick the strongest available cipher.

My goal is having something ready for copy/paste with strongest (reasonable) defaults. Compatible with Kicksecure and/or Whonix-Workstation.
(image you set up a new admin VM and a new remote server)

“You have to read the config, make changes for your needs” usually results in a support nightmare of tons of users keeping lets say some comment in ( ) behind config keywords and then breaking their sshd server startup which could lock them out from their servers. If we wanted to go that route, better don’t offer any config for copy/paste but just suggestions which settings someone should tune in their sshd config file. That style however is accessible to fewer users.

I was even considering to ship these configs by default with Kicksecure / Whonix if they matured but regressions that could cause seems high. Not cool to lockout someone from accessing their ssh remote server because package enforced strong ciphers. Fixable for ssh client config but a shock. For server config, a config update by package could lock out users from their servers.

1 Like

@Patrick
I think your proposal is very rational, and I like it. It keeps it secure and not confusing which is important, especially for someone who maybe does not know a lot about ssh.
I have a ssh_config file which I put together, here it is; see if you like it. The user can have this file in either /home/user/.ssh/config or if using it as a systemwide configuration with client and server on same machine /etc/ssh/ssh_config. The options are the same for either location. If both the openssh-client and openssh-server are going tot be included, then having it in /etc/ssh makes sense. If someone has a specific requirement, a config in /home/user/.ssh/config can be generated
Here is the systemwide /etc/ssh/ssh_config

Host *
ForwardAgent no
AddressFamily inet
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
Port 22
Protocol 2
PubkeyAuthentication yes
StrictHostKeyChecking ask
HashKnownHosts yes
User user
Host host
SendEnv LANG LC_*
# Ciphers chacha2-=poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr
# MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-512,hmac-sha2-256-etm@openssh.com
# KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384

That could be a reasonable config. The ciphers,macs.and kex are there so the user can see them, but commented to allow the client and server to negotiate their own (which would be the chacha20, hmac-sha2-512-etm, and curve25519 anyway especially since the packages are up to date)
I tested this file on Debian (virtual machine) and it worked. I did have to create the “authorized_keys” and “known_hosts” files in ~/.ssh, but I had to do that on Ubuntu also.

1 Like

@Patrick
I just remembered this about the openssh-server: when I first installed it from Debian, I ran a sudo dpkg-reconfigure openssh-server in order to re-generate the included keys that come with the server in /etc/ssh. The regular install includes a rsa, ecdsa and ed25519 server keys. After running the command, I removed the rsa and ecdsa keys also. Is that command necessary for our purposes? In other words will all the openssh-servers have the same keys or are they different by default for each machine?

1 Like

Shouldn’t be required unless you want to change keys.

Generated locally. Should be different for each machine. Otherwise Debian would have a severe security issues.

I was wondering if we should advice that but I guess server setting HostKey /etc/ssh/ssh_host_ed25519_key and client setting IdentityFile ~/.ssh/id_ed25519 should be sufficient.

No surprise for me.

1 Like

Side note might be added in post-quantum page or SSH or no where:

OpenSSH version OpenSSH 8.0/8.0p1 (2019-04-17) and above:

  • ssh(1), sshd(8): Add experimental quantum-computing resistant
    key exchange method, based on a combination of Streamlined NTRU
    Prime 4591^761 and X25519.

might be useful for the wiki or later reference.

2 Likes

Renewed KexAlgorithms keys in client and server from curve25519-sha256SSH@libssh.org to curve25519-sha256SSH

Evidence: Here

and added to sshd/server “DebianBanner no” to remove debian version from ssh banner.

Evidence: Here

Thanks to Gabe from grepheneos for pointing that out.

1 Like

OpenSSH 8.5 released on (2021-03-03) with this update for pqcrypt

https://www.openssh.com/txt/release-8.5

  • ssh(1), sshd(8): update/replace the experimental post-quantum
    hybrid key exchange method based on Streamlined NTRU Prime coupled with X25519.

The previous sntrup4591761x25519-sha512@tinyssh.org method is replaced with sntrup761x25519-sha512@openssh.com.

Per its designers, the sntrup4591761 algorithm was superseded almost two years ago by sntrup761.

(note this both the updated method and the one that it replaced are disabled by default)

1 Like

After reading some man pages you might consider to add the following to Secure Shell (SSH) :

https://manpages.debian.org/bullseye/openssh-client/ssh_config.5.en.html#CheckHostIP
https://manpages.debian.org/bullseye/openssh-client/ssh_config.5.en.html#FingerprintHash
https://manpages.debian.org/bullseye/openssh-client/ssh_config.5.en.html#ForwardX11Trusted
https://manpages.debian.org/bullseye/openssh-client/ssh_config.5.en.html#GatewayPorts
https://manpages.debian.org/bullseye/openssh-client/ssh_config.5.en.html#VerifyHostKeyDNS

CheckHostIP yes
FingerprintHash sha256 (the default)
ForwardX11Trusted no (upstream default, not in debian)
GatewayPorts no (the default)
VerifyHostKeyDNS ask (the default is no)

I also found some interesting things here: OpenSSH
for the sshd_config, especially:

# Use kernel sandbox mechanisms where possible in unprivileged processes
# Systrace on OpenBSD, Seccomp on Linux, seatbelt on MacOSX/Darwin, rlimit elsewhere.
UsePrivilegeSeparation sandbox

Also the LogLevel part.

1 Like