I have become a kind of fan of Cloud At Cost. Their one-time-fee servers and easy build process is great for spinning up test machines. I would hardly recommend running anything that I would consider “production” or mission critical on a cloud at cost VM, but it is a cheap, quick, and simple way to spin up boxes to play with until you are ready for more expensive/permanent hosting (like with Digital Ocean or Amazon). Spinning up a new box means securing SSH. So here is my guide.
The major problem with a hosted server of any kind is drive-by scans. There are folks out there that scan for huge swaths of the Internet looking for vulnerable machines. There are two basic varieties: scanning a single host for all vulnerabilities, and scanning a large number of hosts for a specific vulnerability. A plain box should really only be running SSH, so that is the security focus of this post. There should also be a firewall running, that rejects connections on all ports except the services you absolutely need.
It should be noted that Your security measures don’t necessarily have to be top notch, your box just has to be less convenient than the next host on the scanners’ lists. It’s not hard to scan a large subnet and find hosts to hammer on. Drive-by scans are a numbers game; it’s all about the low hanging fruit. With C@C, it’s a question of timing. You have to get onto the box and lock it down quickly. Maybe I’m just being paranoid, but I have had boxes that I didn’t log in to right after spinning them up and I have seen very high CPU utilization on them when they aren’t really running anything, which leads me to believe that the host has been compromised. Also, beware that the web-based stats can be wildly inaccurate.
This guide will only lock down SSH. If you are running a web server, this guide will not lock down the web server. If you are running Asterisk, this guide will not lock down Asterisk. All this guide will do is shore up a couple of vulnerabilities with SSH. I recommend running these steps *BEFORE* installing anything on your VM.
My use case for Cloud At Cost is something like this: There are times when I need a box that is easier to get to than hosting a box on my home network, but doesn’t really justify the monthly cost of running a server on Digital Ocean or Amazon. For me, I spend a lot of time working all night inside a very restrictive corporate network, so it’s hard to get access to my stuff at home especially since Team Viewer is compromised. C@C is cheap and easy, which probably means it’s a playground for scammers and other bad actors. This means it’s a good idea to lock down your box before you do anything useful with it.
You can get started with C@C for around $35, but if you follow them closely, you can catch some of their discount deals and get a very low end developer box for around $10. I took advantage of a few of these promotions and now I have a bucket of resources at my disposal for all of my tinkering needs. Also, if your box starts to misbehave (loads of network traffic, high cpu utilization, etc.) it’s probably compromised, so just torch it and build a new one.
You can learn about the basics of the Cloud At Cost panel here, the info will be useful later on:
Once you have signed up with C@C, bought some resources, and fired up your Linux VM, it’s time to do some housekeeping. I prefer Debian, and it’s what I am using in this guide, but it doesn’t really matter what you choose.
As soon as the box is up, log in with SSH, using the root password given in the information button. I use putty*, because most of my time in front of a computer is spent working or gaming, so I use Windows a lot. I know it upsets a lot of folks to hear that, but hey, those folks can feel secure in knowing that their “Unix Beards” are mightier than mine.
The very first thing that I do is change the root password. Make like 30 or more random characters. You shouldn’t actually need to type it in after this point, but keep it somewhere encrypted just in case. I also comment out the non-us repo that C@C Debian machines are still pointed to in sources list:
Just locate the line that begins with “deb http://non-us.debian.org” and put a # in front of it. On a C@C Debain 8 box, it should be the first line.
With that pesky non-US entry removed, you are clear to update your packages:
I also run these commands from the Nerd Vittles blog to make sure the password doesn’t revert to the Cloud At Cost root password:
sed -i '/exit 0/d' /etc/rc.local
echo killall plymouthd >> /etc/rc.local
rm -f /etc/rc3.d/S97*
echo "exit 0" >> /etc/rc.local
I don’t know if they are strictly necessary, but the dudes at Nerd Vittles recommend it, and they spend waaaay more time doing this stuff than I do, so there you have it.
After that, it’s time to install fail2ban, and then create a non-root user:
apt-get install fail2ban
Hopefully, in a few minutes fail2ban will be made superfluous by our additional security measures. In the meantime it will stop brute force attempts. Some of my hacker buddies change the default port for SSH to throw off driveby scans, but the restrictive corporate network I mentioned before doesn’t like arbitrary ports, so that’s a hard no in this case.
Enable Sudo for a Non-Root User
To start implementing our security measures, we will install sudo, add ‘steve’ (our non-root user) to the sudo group, and then make sure steve has the right permissions in the sudoers file:
apt-get install sudo
adduser steve sudo
At this point the /etc/sudoers file should open in the Nano next editor. I know I should be using vi, but I am too busy #YOLOing to do that Unix Beard crap. 🙂
Press ‘ctrl+w’ to open the search box, and type ‘%sudo’ to find the permissions line.
Press ‘ctrl+k’ to cut the ‘%sudo ALL=(ALL:ALL) ALL’ line, and then ‘ctrl+u, ctrl+u’ (hold ctrl and press ‘u’ twice) to paste the line in twice.
Edit the second line to read ‘steve ALL=(ALL:ALL) ALL’ and press ‘ctrl+x’ to exit, and press enter to save.
Setting up sudo is important because we are going to disable root logins here in a minute, but first we are going to set up SSH Keys for logins and then disable clear text logins. SSH does use clear text passwords, but it passes them through an encrypted tunnel. This means that while your password isn’t likely to be sniffed, it could be guessed or brute forced. Using SSH keys means you have to have the right private key to match with a public key on the server. But before we can do any of that, we need to test the new non-root account by logging in with it.
Once you are logged in as steve, test sudo:
Which should return ‘root’.
Securing SSH with Asymmetric Keys
Once the non-root account is working and sudo-ing, we can proceed to lock down SSH with public+private key pairs. I will explain how to do this with putty for Windows, but it’s actually way easier to do this with Unix.
The first step is to make sure you have puttygen.exe handy. Download it and launch it, change the bits for your keys to 4096 (in the lower right corner) then click the ‘Generate’ button.
Wiggle the mouse around for a bit, and in a minute or so you will see your public key, with a key comment and blanks for your passphrase. You don’t have to change the comment, or enter a passphrase, but I recommend it. I like to change the comment to match the username and server (‘firstname.lastname@example.org’ in the screenshot below), since I have lots of different keys. The passphrase keeps things safe in case your private key file falls into enemy hands.**
At this point, you may be tempted to use the same passphrase for your private key as you use for your non-root user account. This is a bad idea, because your non-root password is now basically your root password. Do yourself a favor and use two completely different passwords.
Next, click ‘Save private key’ and save the resulting .ppk file in a safe location, but don’t close the puttygen window just yet. If you use multiple computers, putty will let you re-use your private key file between Windows machines, if that’s what you’re into. SSH on Linux may, but it will not let you use a puttygen file in a Linux system. (Based on that one time I tried it and it didn’t work for me.) So just keep that in mind.
Also, it’s no big deal to have multiple private/public key pairs on the same server. You can use a different pair for each client computer, which is probably safer and more convenient than using a shared key pair. If you lose access to a client machine for whatever reason, you can just delete the public key off of the server and that machine won’t be able to connect to your server.
Leave your puttygen window up and switch back to your putty/SSH window. Create a .ssh folder and a key file for SSH, then a text file to store your keys:
Paste the Public Key text in the top of the puttygen window onto a single line in the file. This will be a Very Large Line Of Text(tm) (VLLOT). The VLOTT should begin with ‘ssh-rsa’ and end with ‘rsa-key-yyyymmdd’ where yyyymmdd is the date you created the key. Sometimes the key comment (email@example.com in the example below) is the last bit of text. I haven’t quite nailed down why that is, presumably an order of operations thing. Anyway, be sure that the VLOTT begins with ssh-rsa, or you didn’t grab all the text in the public key.
Save and close the file (‘ctrl+x’ and then ‘enter’) and then set the permissions for the file:
chmod 600 ~/.ssh/authorized_keys
Now exit your ssh session, and reopen putty. You need to set the IP address of your server as the hostname. I prefer this to host names because DNS can’t always be trusted. Give your session a useful name.
Under ‘Connection -> Data’ add the username for your non-root account. In this example, I named my account ‘steve’.
Under ‘Connection -> SSH -> Auth’ browse to the safe place you saved your private key. You pasted your public key onto the server, and you have your private key stored on your computer. You will want to keep the private key file safe because if you lose it you have to set up a new pair while logged in at the console, which is a total pain. I keep mine in Dropbox, but I keep them secured with a passphrase.**
Now go back to Session and save your session profile. Henceforth you can connect simply by double clicking ‘steve’s server’ under ‘Saved Sessions’.
Now it’s time to test your new key pair. Just double click ‘steve’s server’ and you should be prompted for the passphrase that you set for your private key. Once you enter it, you should be logged in to the server as user ‘steve’. If you were able to log in using your key, you are all set to move on. You are now free to close PuttyGen.
If The Server Rejects Your Key
It’s most likely that you didn’t paste the public key correctly. This is why we left the PuttyGen window open. 🙂
Log in with your non-root username and password (‘steve’ in this example) and open your ~/.ssh/authorized_keys file in nano again:
In the PuttyGen window, make sure that you scroll to the top of the public key text. It should begin with ‘ssh-rsa’. Now click and drag down to the end of the public key text, then right click and select ‘copy’.
In the Putty window, with your authorized_keys file open in nano, delete the incomplete key and paste the complete text of the public key on a single VLLOT.
Save and exit nano, then exit your SSH session and try again.
Also make sure that you changed the permissions of the authorized_keys file:
chmod 600 ~/.ssh/authorized_keys
If your key is still being rejected, generate a new public and private key by clicking the ‘Generate’ button and starting the whole key process over again.
Disable Root and Cleartext Logins
Once your keypair is working, (and you are able to log in with it) it’s time to eliminate root logins and cleartext logins. Some folks will tell you that root logins are fine with SSH because passwords don’t get sent in the clear. While that’s true, ‘root’ is still the one username that is guaranteed to be on every Unix-based machine, so if you are going to brute force an account, this is the one to focus your efforts on. Disabling root logins and clear text logins is all done in the sshd_config file:
sudo nano /etc/ssh/sshd_config
Press ‘ctrl+w’ and search for the word ‘root’. You are looking for this entry:
Change ‘#PermitRootLogin yes’ to ‘PermitRootLogin no’. (uncomment if necessary and change from ‘yes’ to ‘no’.)
Then press ‘ctrl+w’ and search for the words ‘clear text’. You are looking for this entry:
# Change to no to disable tunnelled clear text passwords
Change ‘#PasswordAuthentication yes’ to ‘PasswordAuthentication no’ (uncomment and change from ‘no’ to ‘yes’.)
Once these changes are made, DO NOT LOG OFF OF YOUR SSH SESSION. Once these changes are implemented, it will be hard to log back in to undo anything if you make a mistake. You should have tested and succeeded with your ssh-key based login because we are about to restart the ssh daemon and prevent clear text logins:
sudo systemctl restart ssh
To test ssh logins, connect to the IP of your server with putty using the ‘Default Settings’ profile. Your login attempt should fail because only people with private keys are allowed to the party:
At this point you are far from being hack-proof, but you are a bit more locked down than you were before, and there are always more convenient targets out there 🙂
Hardening web servers is another story, which really isn’t my bag to be honest. There’s a reason that I host my blogs with Google or WordPress 🙂
* Protip: put your putty.exe file in ‘c:\windows\system32’ so you can run putty from the command line or the run line. If you want to be a real hard rock, rename putty.exe to ssh.exe. Did you know putty accepts commandline args? It does, so you can do awesome Unixy shit from the command line like type ‘ssh firstname.lastname@example.org’ to connect to a remote host. It still pops up your connection in the putty window, but it keeps your hands on the keyboard. 🙂
** Another Protip: not setting a passphrase is handy for automating ssh connections, especially if you want to move files back and forth with ‘scp’ or mess with tunneling via local and remote ports. I haven’t found a decent scp command line app for Windows, other than the Unix utils in CygWin.