There's a few weird things, but it's mostly okay.
Do not trust the firewall on the bastion host, if an attack can get into the bastion host, they can disable the firewall, so it cannot be used to limit egress. It's better than nothing, but consider using a firewall that's managed on a via a separate management network. I do agree that you should only allow SSH from a few known IPs.
Limiting the number of users is weird, and not recommended. Create all the accounts you need to provide individual accounts for the staff that need to access the bastion host, you will need that as things like HIPAA require named accounts for auditing. None of the accounts need any privileges other than the most basic. Users do not need sudo/root privileges on a jump host.
Other than those two complains, it's good recommendations.
A final recommendation: If you use AWS though, consider using Session Manager instead of SSH and drop the bastion host. You can still connect using the SSH command, using proxy command in OpenSSH, but no public IP or bastion host is required.
Twice I've seen Bastion Hosts compromised. Both times it practically gave the attackers the highest access. In one case it basically hid where the attack came from (compromised logs and all). In another it let them hijack an admin's password by reading his sudo.
IMHE, Bastion Hosts suck.
If you are forced to use one, send logs to a safer one-way storage encrypted and put tampering triggers everywhere you can in the Bastion Host. Also make sure you log outgoing connections. And make sure you can easily match incoming to outgoing.
If you absolutely have to use sudo on the Bastion Host force it to OTP only. Or if absolutely not possible, use 2FA, but this is a risk as something somewhere might not be properly protected and the password will leak. But the better way would be to have the bastion host run on some read-only image and not letting it upgrade or do any admin task at all. Maybe even remove admin users, SSH, the whole lot.
And related, do not have a single account with god-like access to everything. Isolate permissions. This is probably the hardest to get OK'd but it's the classic SPOF where they got you by the balls.
It does not seem to be mentioned here. But, my #1 hardening suggestion is install the Tripwire IDS (Intrusion Detection System). It is probably the best thing you could ever do for yourself as a system administrator. It integrity checks the entire file system. If anything happens to your system that you didn't authorize you're notified of it immediately. After initial install it is important to minimize and exclude false positives so that you end up with a system that rarely changes in ways you don't expect or can at least explain.
Another really useful tool is logwatch.
I actually caught an intruder this way hijacking my system several years ago. They removed rkhunter, chkrootkit and a variety of log files. And, modified lines in the last logged in users log. But, a combination of logwatch and tripwire caught it.
https://opensource.com/article/18/1/securing-linux-filesyste...
The premise sounds iffy ("SSH bastion hosts are an indispensable security enforcement stack for secure infrastructure access").
Every time you build some infrastructure, you expend scarce resources like engineering effort (=opportunity cost), time, money, and complexity by adding moving parts to your christmas tree of technology. You should always critically evaluate what's the most low hanging fruit you can invest in for a given end goal (eg improving security) considering the complexity costs. SSH bastions can be worth implementing in some situations, but not top of the list in many cases.
The next sentence starts talking about "security compliance standards" - you sometimes have to submit to doing stuff for reasons of ticking boxes, but it's important to remember when you're doing what's best for security and when you're going through motions mainly to tick boxes for someone else.
Good writup. One thing I would add for bastions if you wanted to harden them would be to disable session multiplexing if you are using MFA/2FA.
MaxSessions 1
Another risk with a bastion model is port forwarding. As an organization you have to decide what is appropriate for that bastion. Unrestricted forwarding? Restricted? Denied?
AllowAgentForwarding no
AllowTcpForwarding yes
PermitOpen 192.168.1.2:22
If your bastion is also used for automation to drop files into a staging area, you can limit that automation to file transfers and even limit what it may do with files. This prevents the automation from having a shell or performing port forwarding.
The keys should be outside of the home directories to prevent malicious tools from appending additional authorized_keys into the account. Make use of automation to manage key trusts and add a comment to keys to map them to an internal tracking system like Jira. This assumes your MFA/2FA is excluding specific accounts or groups via PAM and permitting the use of ssh keys with specific groups or accounts.
AuthorizedKeysFile /etc/ssh/keys/%u
Match Group sftpusers
Banner /etc/ssh/banner_sftp.txt
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
GatewayPorts no
ChrootDirectory /data/sftphome/%u
ForceCommand internal-sftp -l DEBUG1 -f AUTHPRIV -P symlink,hardlink,fsync,rmdir,remove,rename,posix-rename
AllowTcpForwarding no
AllowAgentForwarding no
Another thing mentioned in the article is iptables. In a PCI environment one may want to also have explicit outbound rules using the owner module to limit what users or groups are permitted to ssh out. So if your organization have a group of people allowed to use this host as a bastions, then one could write a rule like
iptables -I OUTPUT -m owner --gid-owner devops -p tcp --dport 22 -d 192.168.0.0/16 -j ACCEPT
Lastly I would add that bastions should have as minimal an OS install possible and have SELinux enforcing. Actions denied by SELinux should go to a security operations center after you spend some time tuning out the noise and false positives.
[1] - https://man7.org/linux/man-pages/man8/sftp-server.8.html
It recommends:
# Configure idle time logout
ClientAliveInterval <value in seconds>
A superset of these best practices in the article would be CIS benchmarks. Collectively agreed on by industry leaders and provide extensive resources that span the gamut of cloud, networking, and storage infrastructure.
CIS supported technologies: https://www.cisecurity.org/cis-benchmarks
CIS Audit AWS infra: https://github.com/toniblyx/prowler
Better to be proactive than reactive :^)
If you decide to use a bastion host, why not use a security-focused OS like OpenBSD?
The "how to grant and manage access to resources" issue is still unsolved in my opinion. There is a middle ground somewhere between raw bastions and managed access services or open VPNs that could be filled.
There are a few different players in this space, but the one to watch is Boundary by Hashicorp.
https://www.boundaryproject.io/
Basically managed authenticated proxy connections to any resource you could possibly need. Still young, so it's missing auditing and some of the convenience features, but give it a year and it will be a compelling open source competitor.
Teleport is great, but their centralized model is not suitable for all situations.. and the pricing (at least for kubernetes) leaves a lot to be desired.
There is also StrongDM, which is very similar with a better pricing model.
Newbie here to VPCs, bastion hosts, VPEs, etc. After spent a while on the topic, some questions arise that you might find redundant or can answer.
I am wondering why we need to configure this many steps as outlined in the article, and in general. What is the point of Teleport in the first place? Why is there no managed service that takes care of all of that, with me focusing on just deploying an app and running it in the VPC.
Can’t 99% of the use cases be put in a template and managed by a service provider, including the security?
Can someone explain to me the benefits of limiting the IPs that can SSH into the bastion? It seems to me the main thing that's protecting against are misconfigurations of SSH (accidentally letting root log in with no password or something) or a zero day in SSH but I'm not convinced by either.
Could someone do me a solid and explain best security practices around bastion hosts and vpn?
e.g. - would you still require users connected to the vpn to go through a bastion host? - would you ever run bastion/vpn through the same box? - are there preferred access use cases for each?
very nice writeup - one of the better ones i have seen. you can go a step further and eliminate open inbound port 22 (make the sshd server 'dark' to the network) with open source solutions like this:
https://ziti.dev/blog/zitifying-ssh/
disclosure: we build SaaS on top of OpenZiti (the open source) so are opinionated in this domain. and, to be clear, the above is just one layer...other layers of security still apply.
Maybe I missed it but did they cover logging all keystrokes entered by users over the bastion? (In the case where you need to log into it first vs merely doing port forwarding)
People still use Bastion hosts?
I'm trying to grok why they're better than SOCKS5 proxies... Is it because they provide shell access and a larger attack surface? ;-)
SSH Bastion Best Practices --> Don't use one
Some great tips here, take my upvote!
No mention of sshguard or fail2ban?
Or you could just use strongdm!
Wireguard to SSH
I think this article has come up before? Either way, it's a quirky thing for Gravitational to post, since their flagship project --- Teleport --- basically eliminates bastion servers altogether (you might think of it as an API-controlled self-contained bastion server). Teleport is free, and worth checking out: it solves a bunch of SSH management problems, not just controlling access, but also linking SSH access to SSO, running fleet-wide commands selectively, and generating transcripts of SSH sessions.
Teleport is kind of big and sprawling. But they've repeatedly contracted Doyensec to do assessment work for it, and Doyensec is a fantastic firm. I think parked behind Tailscale, so none of your SSH infra is exposed to the Internet to begin with, it's a pretty great solution, and I'd do that again before I ever hand-tooled an SSH bastion host again.