I have been writing a lot about this Raspberry Pi iPad server project and Mosh pretty much seemed purpose-built to solve the “ssh on a mobile device that hates background processes and open connections” problem.
I was doing some clever shit with screen, but I gave mosh a go and it’s dead simple to configure and use with blink.
You literally just install the mosh server on the Pi:
sudo apt install mosh
And the Mosh client is built right into Blink, which is cool. I guess.
In the last installment of this series, I discussed setting up the Proxmox VE hosts. Until now, you could do most of this configuring in triplicate with MobaXTerm. Now you can still use it to multicast, just be sure to disable it when you have to customize the configs for each host. This part of the process is a lethal combination of being really repetitive while also requiring a lot of attention to detail. This is also the point where it gets a bit like virtualization-inception: VirtualBox VMs which are PVE hosts to PVEVMs.
Network Adapter Configuration
I did my best to simplify the network design:
I have built this cluster a few times and my Ethernet adapter names (enp0s3, enp0s8, and enp0s9) have always been the same. That may be a product of all the cloning, so YMMV. Pay close attention here because this can get very confusing.
Open the network interface config file for each PVE host:
nano /etc/network/interfaces
You should see the entry for your first Ethernet adapter (the bridged adapter in VirtualBox), followed by the virtual machines' bridge interface with the static IP that you set when you installed Proxmox. This is a Proxmox virtual Ethernet adapter. The last two entries should be your two host only adapters, #2 and #3 in VirtualBox. These are the adapters that we need to modify. The file for prox1 probably looks like this:
nano /etc/network/interfaces
auto lo
iface lo inet loopback
iface enp0s3 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.101
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports enp0s3
bridge_stp off
bridge_fd 0
iface enp0s8 inet manual
iface enp0s9 inet manual
Ignore the entries for lo, enp0s3, and vmbr0. Delete the last two entries (enp0s8 and enp0s9) and replace them with this:
#cluster network
auto enp0s8
iface enp0s8 inet static
address 192.168.2.1
netmask 255.255.255.0
#migration network
auto enp0s9
iface enp0s9 inet static
address 192.168.3.1
netmask 255.255.255.0
Now repeat the process for prox2 and prox3, changing the last octet for the IP's to .2 and .3 respectively. When you are finished your nodes should be configured like so:
Save your changes on each node. Then reboot each one:
shutdown -r now
Network Testing
When your PVE hosts are booted back up, SSH into them again. Have each host ping every other host IP to make sure everything is working:
ping -c 4 192.168.2.1;\ ping -c 4 192.168.2.2;\ ping -c 4 192.168.2.3;\ ping -c 4 192.168.3.1;\ ping -c 4 192.168.3.2;\ ping -c 4 192.168.3.3
The result should be 4 replies from each IP on each host with no packet loss. I am aware that each host is pinging itself twice. But you have to admit it look pretty bad ass.
Once you can hit all of your IP's successfully, now it's time to make sure that multicast is working properly. This isn't a big deal in VirtualBox because the virtual switches are configured to handle multicast correctly, but it's important to see the test run so you can do it on real hardware in the future.
First send a bunch of multicast traffic at once:
omping -c 10000 -i 0.001 -F -q 192.168.2.1 192.168.2.2 192.168.2.3
You should see a result that is 10000 sent with 0% loss. Like so:
192.168.2.1 : unicast, xmt/rcv/%loss = 9406/9395/0%, min/avg/max/std-dev = 0.085/0.980/15.200/1.940 192.168.2.1 : multicast, xmt/rcv/%loss = 9406/9395/0%, min/avg/max/std-dev = 0.172/1.100/15.975/1.975 192.168.2.2 : unicast, xmt/rcv/%loss = 10000/9991/0%, min/avg/max/std-dev = 0.091/1.669/40.480/3.777 192.168.2.2 : multicast, xmt/rcv/%loss = 10000/9991/0%, min/avg/max/std-dev = 0.173/1.802/40.590/3.794
Then send a sustained stream of multicast traffic for a few minutes:
omping -c 600 -i 1 -q 192.168.2.1 192.168.2.2 192.168.2.3
Let this test run for a few minutes. Then cancel it with CTRL+C.
The result should again be 0% loss, like so:
root@prox1:~# omping -c 600 -i 1 -q 192.168.2.1 192.168.2.2 192.168.2.3 192.168.2.2 : waiting for response msg 192.168.2.3 : waiting for response msg 192.168.2.3 : joined (S,G) = (*, 232.43.211.234), pinging 192.168.2.2 : joined (S,G) = (*, 232.43.211.234), pinging ^C 192.168.2.2 : unicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.236/1.488/6.552/1.000 192.168.2.2 : multicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.338/2.022/7.157/1.198 192.168.2.3 : unicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.168/1.292/7.576/0.905 192.168.2.3 : multicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.301/1.791/8.044/1.092
Now that your cluster network is up and running, you can finally build your cluster. Up to this point, you have been entering identical commands into a SSH sessions. At this point, you can stop using the multi-exec feature of your SSH client.
First, create the initial cluster node on Prox1, like so:
root@prox1:~# pvecm create TestCluster --ring0_addr 192.168.2.1 --bindnet0_addr 192.168.2.0
Then join Prox2 to the new cluster:
root@prox2:~# pvecm add 192.168.2.1 --ring0_addr 192.168.2.2
Followed by Prox3:
root@prox3:~# pvecm add 192.168.2.1 --ring0_addr 192.168.2.3
One final configuration on Prox1 is to set the third network interface as the dedicated migration network by updating the datacenter.cfg file, like so:
root@prox1:~# nano /etc/pve/datacenter.cfg keyboard: en-us migration: secure,network=192.168.3.0/24
Now that the cluster is set up, you can log out of your SSH sessions and switch to the web GUI. When you open the web GUI for Prox1 (https://192.168.1.101:8006) and you should see all 3 nodes in your TestCluster:
Now you can manage all of your PVE Hosts from one graphical interface. You can also do cool shit like migrating VMs from one host to another, but before we can do that we need to set up some PVEVMs. There are more things to set up, but to see them in action, we need to build a couple of PVEVMS to work with, which I will cover in the next installment: Building PVE Containers.
In the last installment of this series, I discussed setting up the Proxmox VE hosts in VirtualBox. At this stage in the exercise there should be 3 VirtualBox VMs (VBVMs) running, in headless mode. These VBVMs should be running Proxmox VE, ready to host their own VMs (PVEVMs).
Before you can set up the cluster, storage replication, and high availability, you need to do a bit of housekeeping on your hosts. In this post, I will go over those steps making sure that the hosts are up to date OS wise and that your storage is properly configured for later use. Most of these steps can be accomplished via the Web UI, but using SSH will be faster and more accurate. Especially when you use an SSH client like SuperPuTTY or MobaXTerm that lets you type and paste into multiple terminals at the same time.
Log in as root@ip-address for each PVE node. In the previous post, the IPs I chose were 192.168.1.101, 192.168.1.102, and 192.168.1.103.
I don’t want to bog this post down with a bunch of Stupid SSH Tricks, so just spend a few minutes getting acquainted with MobaXTerm and thank me later. The examples below will work in a single SSH session, but you will have to paste them into 3 different windows like a peasant, instead of feeling like some kind of superhacker:
Step 1 – Fix The Subscription Thing
No, not the nag screen that pops up when you log into the web UI, I mean the errors that you get when you try to update a PVE host with the enterprise repos enabled.
All you have to do is modify a secondary sources.list file. Open it with your editor, comment out the first line and add the second line:
nano /etc/apt/sources.list.d/pve-enterprise.list
# old and busted
#deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise
# new hotness
deb http://download.proxmox.com/debian/pve stretch pve-no-subscriptionSave the file, and run your updates:
apt-get update; apt-get -y upgradeWhile you are logged in to all 3 hosts, you might as well update the list of available Linux Container templates:
pveam updateFinally, if you set up your virtual disk files correctly according to the last post, you can set up your ZFS disk pool:
root@prox1:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 32G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 31.5G 0 part
├─pve-swap 253:0 0 3.9G 0 lvm [SWAP]
├─pve-root 253:1 0 7.8G 0 lvm /
├─pve-data_tmeta 253:2 0 1G 0 lvm
│ └─pve-data 253:4 0 14G 0 lvm
└─pve-data_tdata 253:3 0 14G 0 lvm
└─pve-data 253:4 0 14G 0 lvm
sdb 8:16 0 64G 0 disk
sdc 8:32 0 64G 0 disk
sr0 11:0 1 642M 0 rom
root@prox1:~#
Assuming you see those two disks, and they are in fact ‘sdb’ and ‘sdc’ then you can create your zpool. Which you can think of as a kind of software RAID array. There’s waaaay more to it than that, but that’s another post for another day when I know more about ZFS. For this exercise, I wanted to make a simulated RAID1 array, for “redundancy.” Set up the drives in a pool like so:
zpool create -f -o ashift=12 z-store mirror sdb sdc
zfs set compression=lz4 z-store
zfs create z-store/vmdata
In a later post we will use the zpool on each host for Storage Replication. The PVEVM files for each of your guest machines will copy themselves to the other hosts at regular intervals so when you migrate a guest from one node to another it won’t take long. This feature pairs very well with High Availability, where your cluster can determine if a node is down and spin up PVEVMs that are offline.
Now that your disks are configured, it’s time to move on to Part 3: Building The Cluster.
I use this site for a number of reasons. One them is to keep a little diary of things that I have figured out so that I can reference them later. The problem is that those little discoveries are buried in rambling posts about why I chose to do something. So here is where I talk about setting up a dynamic port to use as a SOCKS proxy for securing web traffic, local ports for accessing resources inside of closed network, and remote ports for making resources inside of a closed network publicly available on the Internet.
I have given 2600 talks about SSH tunnels but I really don’t have a permanent record of my various [mis]uses of SSH, so I thought I would put them all in a post for future reference. I have written about securing SSH with asymmetric encryption keys, but there are many more things that you can do with SSH.
IMPORTANT NOTE: – none of these tricks work unless your host has TCP port forwarding enabled. This option is usually enabled by default, but you should double check your /etc/ssh/sshd_config file:
nano /etc/ssh/sshd_configThen make sure that the option is uncommented and enabled:
AllowTcpForwarding yesAlso, if all this putty crap looks tough to do, just remember that all of this is way easier with UNIX. Particularly using the -J option for jumping SSH connections.
Dynamic Ports: slip past content filters with this one weird trick
This is the go-to use for an SSH tunnel. University and corporate networks are often overseen by petty tyrants who limit access to sites of questionable content, or sites that are prone to eating bandwidth, like social and streaming media. Ironically, most of these networks allow outbound SSH. SSH kills content filters dead.
You can create a SOCKS proxy using an SSH connection by creating a dynamic port. You can then point your browser to use your local address (127.0.0.1) as a SOCKS proxy to smuggle all of your browser traffic thru the SSH tunnel.
I like using Firefox specifically for this task because Chrome or IE does dumb things when you mess with the proxy settings. Or rather, they did dumb things that one time I used them to tunnel 10 years ago and then decided on using FireFox. If you want to also tunnel your DNS queries, in Firefox type “about:config” in the browser bar and then find the setting “network.proxy.socks_remote_dns” and set it to “true”. There’s probably a more modern way to to that, but that’s the tried and true way. Tunneling DNS isn’t strictly necessary, but it does help you stay under the radar on a restrictive network.
It should be noted that once you tunnel your browser traffic through the tunnel, you are likely to lose access to intranet sites on your local subnet. For this and a number of other reasons, I like to run Brave (Chrome) for my normal stuff, and Firefox for the tunneled stuff. Running two browsers at the same time seems wasteful, but it saves a bunch of headaches.
On Windows, you can drop the port one of three ways:
If you are into radio buttons and text boxes, you can configure PuTTY to open one every time you connect. On the tree control to the left, click Connection -> SSH -> Tunnels. You’ll need to select dynamic and enter a source port (5555 in the picture to the right) and click ‘Add’. Your new port should appear in the list above as ‘D5555’. Be sure to go back to your session and click ‘Save’ of you want the port to be created every time you open the session. As long as you are messing with your PuTTY session, you might as well set your terminal colors to green text so you look like a real hacker.
If using the GUI makes you feel like a wimp, you can just script it with a handy batch file:
putty -D 5555 user@ssh.server.comYou can also feel like a real UNIX badass by copying putty.exe to c:\Windows\System32 and renaming it to ssh.exe so you can kick off your session from DOS like a real console cowboy.
OR you can dump the whole interactive shell pretense and use plink.exe to make your connection and drop your ports without the whole pesky PuTTY window getting in your way:
plink -D 5555 user@ssh.server.comPlink is bascially PuTTY with no window. It functions basically the same in all other respects.
If you are using a Unix or Linux workstation, you can set up your dynamic port with a similar syntax:
ssh -C -D 5555 user@ssh.server.comNote: the -C switch compresses traffic going thru the tunnel, theoretically increasing network speeds through the tunnel.
You can use SSH to secure more than just your web browsing. You can use it to secure pretty much any TCP connection. I like using it to secure notoriously insecure VNC and X sessions.
Another use is to get around port restrictions. Some networks may allow outbound SSH, but only on port 22. Some home ISP’s get shitty about running servers on reserved ports like 22. This means you have to forward some bunk port on your home router, like 62222 to the SSH server on your home network. I used to do this when I had Time Warner cable. The problem would get worse when I was trying to connect remotely from a restrictive network that only let SSH out on port 22.
To get around this problem, I would have to SSH to a public access Unix system like the Super Dimensional Fortress on port 22 and then drop a local port that forwarded to the bunk SSH port on the IP of my home router. When I did that with different windows that had different text colors it looked like I was on CSI: Hacker Girl back tracing the killer’s IP address.
The setup is pretty much the same as for the dynamic port, only you have to specify a destination IP and port as well. The idea is to connect to a remote host that is on the other side of a restrictive firewall and using the SSH tunnel to make something accessible to your local your local network. It forwards all traffic to your local port to the remote destination thru the tunnel. In the example above it was SSH but it could be RDP or any other TCP connection. I’ll use RDP (port 3389) in the example below.
To tunnel a Microsoft Remote Desktop session through SSH using the PuTTY gui, use the tree control to the left, click Connection -> SSH -> Tunnels. You’ll need to select local and enter a source port (13389 in the picture to the right), set the destination address or host+domain name and the port (192.168.1.10:3389 in the picture to the right) and click ‘Add’. 13389 will be the port on your workstation that is now connected to the RDP port on the remote network (3389). Your new port should appear in the list above as ‘L13389 192.168.1.10:3389’. Be sure to go back to your session and click ‘Save’ of you want the port to be created every time you open the session. In your RDP client, you would connect to 127.0.0.1:13389.
If you are scripting this setup, use the -L switch along with your source port, destination IP/host and the destination port. Using the scenario from above, you forward local port 13389 to 192.168.1.10 port 3389 like this, where your SSH username is ‘alice’ and your home network’s dynamic DNS hostname is casa-alice.dynamic.DNS:
putty -L 13389:192.168.1.10:3389 alice@casa-alice.dynamic.DNSAnd finally, the syntax is the same with plink:
plink -L 13389:192.168.1.10:3389 alice@casa-alice.dynamic.DNSYou can actually specify multiple local ports to remote destinations. I do this with PuTTY to get direct access to the web interface on my Proxmox cluster and to RDP to a Windows host using just the one tunnel and without having to mess with my proxy settings.
Remote Ports: SSH kills firewalls dead.
In the local port scenario, you are connecting to a remote host behind a firewall and using the SSH tunnel to make a host inside the remote firewall accessible to your local your local network. You can also do the opposite, which is to connect to a remote host outside of your firewall and use the SSH tunnel to make a host inside your local firewall accessible to either the Internet or to hosts on a remote network.
You do this by dropping a port at the other end of the tunnel. On the remote host. The obvious use is to temporarily punch a hole in the local firewall to expose an internal web server to the Internet. If the remote host that you are connecting to is directly connected to the Internet, (like a hosted VM from Cloud At Cost) you can temporarily open a port on the remote server to tunnel traffic to the web server on your internal network.
A more nefarious use for a remote port would be for a leave-behind box (formerly known as a dropbox before the term became a brand name for cloud storage) to phone home from a target network. Basically you build a cheap single board PC, like a Raspberry Pi or a plug server that you physically plug into a network that you plan on hacking testing for security holes. This approach saves a ton of time reverse engineering firewalls to gain access. There are two basic approaches, load up the box with tools and drop it, or use a minimal box as a router/pivot for tools you are running outside the target network.
To do this with the PuTTY gui, it’s basically the same as setting up a local port. Use the tree control to the left, click Connection -> SSH -> Tunnels. You’ll need to select remote and enter a source port (58080 in the picture to the right), set the destination address or host+domain name and the port (192.168.1.10:80 in the picture to the right) and click ‘Add’. You also need to click the check both of the boxes next to “Local ports accept connections from other hosts” and “Remote ports do the same (SSH-2 only)”. Your new port should appear in the list above as ‘R58080:192.168.1.10:80’. Be sure to go back to your session and click ‘Save’ of you want the port to be created every time you open the session.
If you are scripting this setup, use the -R switch along with your source port, destination IP/host and the destination port. Using the scenario from above, you forward remote port 58080 on ssh.server.com to port 80 on your internal home web server with the IP 192.168.1.10 like this:
putty -R 58080:192.168.1.10:80 alice@ssh.server.comAnd finally, the syntax is the same with plink:
putty -R 58080:192.168.1.10:80 alice@ssh.server.comThe only gotcha with scripting your remote port drop with putty/plink is that I don’t think that there is a command line switch for enabling connections from other hosts, so you have to enable the remote port sharing on the SSH server side.
Making Local and Remote Ports Accessible to other hosts
Sharing your remote and local hosts lets you set up your SSH tunnel on one host and then connect to the tunnel from a different host.
In the case of a local port, you could initiate the SSH session on your home Linux server, and then connect to that port from your Windows workstation. This is handy if you are tunneling RDP and you don’t have PuTTY available on your Windows box. Although PuTTY is super portable so it’s dead simple to smuggle it onto the most locked down of Windows machines.
In the case of the remote port, it’s pretty much mandatory for the web server or dropped box use cases. You can still script the connection, you just have to modify your sshd_config file on your SSH server. On a Debian-esque server you do this by using sudo or su to become root and then type:
nano /etc/ssh/sshd_configYou then add the GatewayPorts option. You can put it anywhere in the file, but I prefer to keep it in the first few lines of the file where entries for port configuration are.
# What ports, IPs and protocols we listen for
Port 22
GatewayPorts yesThen save the file and restart SSH:
systemctl restart sshOr on Debian 9, you use ‘service’ to restart SSH:
service ssh restartI am sure that this option is a big security risk, so I recommend a cheap low powered VM dedicated specifically to bouncing SSH connections. I also recommend securing it with SSH keys. If you are looking to script SSH connections that are secured with SSH keys, I recommend not setting a passphrase on your private key. You can include your private key on the putty/plink commandline with the -i switch:
putty -L 13389:192.168.1.10:3389 alice@casa-alice.dynamic.DNS -i c:\path\to\your\key\key.ppkTL;DR: if you just want to set up keys with putty: IDGAF about Cloud At Cost take me to the Putty screenshots. If you are sitting in front of a computer that you don’t have keys set up on, and you are trying to log into a remote server that you have already locked down: I haven’t set up keys on my other computer like an absolute walnut.
Fun with Cloud At Cost
I have become a kind of fan of Cloud At Cost. Their one-time-fee servers and easy build process is great for spinning up test machines. I would hardly recommend running anything that I would consider “production” or mission critical on a cloud at cost VM, but it is a cheap, quick, and simple way to spin up boxes to play with until you are ready for more expensive/permanent hosting (like with Digital Ocean or Amazon). Spinning up a new box means securing SSH. So here is my guide.
The major problem with a hosted server of any kind is drive-by scans. There are folks out there that scan for huge swaths of the Internet looking for vulnerable machines. There are two basic varieties: scanning a single host for all vulnerabilities, and scanning a large number of hosts for a specific vulnerability. A plain box should really only be running SSH, so that is the security focus of this post. There should also be a firewall running, that rejects connections on all ports except the services you absolutely need, but that’s a blog post for another day.
It should be noted that Your security measures don’t necessarily have to be top notch, your box just has to be less convenient than the next host on the scanners’ lists. It’s not hard to scan a large subnet and find hosts to hammer on. Drive-by scans are a numbers game; it’s all about the low hanging fruit. With C@C, it’s a question of timing. You have to get onto the box and lock it down quickly. Maybe I’m just being paranoid, but I have had boxes that I didn’t log in to right after spinning them up and I have seen very high CPU utilization on them when they aren’t really running anything, which leads me to believe that the host has been compromised. Also, beware that the web-based stats can be wildly inaccurate.
This guide will only lock down SSH. If you are running a web server, this guide will not lock down the web server. If you are running Asterisk, this guide will not lock down Asterisk. If you are running MySQL, for the love of god make sure that it’s only accessible from localhost (127.0.0.1) and that is not accessible from the Internet. All this guide will do is shore up a couple of vulnerabilities with SSH. I recommend running these steps *BEFORE* installing anything on your VM.
My use case for Cloud At Cost is something like this: There are times when I need a box that is easier to get to than hosting a box on my home network, but doesn’t really justify the monthly cost of running a server on Linode, Digital Ocean or Amazon. For me, I spend a lot of time working all night inside a very restrictive corporate network, so it’s hard to get access to my stuff at home especially since Team Viewer is compromised. C@C is cheap and easy, which probably means it’s a playground for scammers and other bad actors. This means it’s a good idea to lock down your box before you do anything useful with it, and keep the useful things that you do with it to the bare minimum.
You can get started with C@C for around $35, but if you follow them closely, you can catch some of their discount deals and get a very low end developer box for around $10. I took advantage of a few of these promotions and now I have a bucket of resources at my disposal for all of my tinkering needs. Also, if your box starts to misbehave (loads of network traffic, high cpu utilization, etc.) it’s probably compromised, so just torch it and build a new one.
Getting Started
You can learn about the basics of the Cloud At Cost panel here, the info will be useful later on:
Once you have signed up with C@C, bought some resources, and fired up your Linux VM, it’s time to do some housekeeping. I prefer Debian, and it’s what I am using in this guide, but it doesn’t really matter what you choose.
As soon as the box is up, log in with SSH, using the root password given in the information button. I use putty*, because most of my time in front of a computer is spent working or gaming, so I use Windows a lot. I know it upsets a lot of folks to hear that, but hey, those folks can feel secure in knowing that their “Unix Beards” are mightier than mine.
The very first thing that I do is change the root password. Make like 30 or more random characters. You shouldn’t actually need to type it in after this point, but keep it somewhere encrypted just in case. I also comment out the non-us repo that C@C Debian machines are still pointed to in sources list:
passwd
nano /etc/apt/sources.list
Just locate the line that begins with “deb http://non-us.debian.org” and put a # in front of it. On a C@C Debain 8 box, it should be the first line.
With that pesky non-US entry removed, you are clear to update your packages:
apt-get update
apt-get upgrade
I also run these commands from the Nerd Vittles blog to make sure the password doesn’t revert to the Cloud At Cost root password:
sed -i '/exit 0/d' /etc/rc.local
killall plymouthd
echo killall plymouthd >> /etc/rc.local
rm -f /etc/rc3.d/S97*
echo "exit 0" >> /etc/rc.local
I don’t know if they are strictly necessary, but the dudes at Nerd Vittles recommend it, and they spend waaaay more time doing this stuff than I do, so there you have it.
After that, it’s time to install fail2ban, and then create a non-root user:
apt-get install fail2ban
adduser steve
Hopefully, in a few minutes fail2ban will be made superfluous by our additional security measures. In the meantime it will stop brute force attempts. Some of my hacker buddies change the default port for SSH to throw off driveby scans, but the restrictive corporate network I mentioned before doesn’t like arbitrary ports, so that’s a hard no in this case.
Enable Sudo for a Non-Root User
To start implementing our security measures, we will install sudo, add ‘steve’ (our non-root user) to the sudo group, and then make sure steve has the right permissions in the sudoers file:
apt-get install sudo
adduser steve sudo
nano /etc/sudoers
At this point the /etc/sudoers file should open in the Nano next editor. I know I should be using vi, but I am too busy #YOLOing to do that Unix Beard crap. 🙂
Press ‘ctrl+w’ to open the search box, and type ‘%sudo’ to find the permissions line.
Press ‘ctrl+k’ to cut the ‘%sudo ALL=(ALL:ALL) ALL’ line, and then ‘ctrl+u, ctrl+u’ (hold ctrl and press ‘u’ twice) to paste the line in twice.
Edit the second line to read ‘steve ALL=(ALL:ALL) ALL’ and press ‘ctrl+x’ to exit, and press enter to save.
Setting up sudo is important because we are going to disable root logins here in a minute, but first we are going to set up SSH Keys for logins and then disable clear text logins. SSH does use clear text passwords, but it passes them through an encrypted tunnel. This means that while your password isn’t likely to be sniffed, it could be guessed or brute forced. Using SSH keys means you have to have the right private key to match with a public key on the server. But before we can do any of that, we need to test the new non-root account by logging in with it.
Once you are logged in as steve, test sudo:
sudo whoami
Securing SSH with Asymmetric Keys
Once the non-root account is working and sudo-ing, we can proceed to lock down SSH with public+private key pairs. I will explain how to do this with putty for Windows, but it’s actually way easier to do this with Unix.
The first step is to make sure you have puttygen.exe handy. Download it and launch it, change the bits for your keys to 4096 (in the lower right corner) then click the ‘Generate’ button.
Wiggle the mouse around for a bit, and in a minute or so you will see your public key, with a key comment and blanks for your passphrase. You don’t have to change the comment, or enter a passphrase, but I recommend it. I like to change the comment to match the username and server (‘steve@stevesblog.com’ in the screenshot below), since I have lots of different keys. The passphrase keeps things safe in case your private key file falls into enemy hands.**
At this point, you may be tempted to use the same passphrase for your private key as you use for your non-root user account. This is a bad idea, because your non-root password is now basically your root password. Do yourself a favor and use two completely different passwords.
Next, click ‘Save private key’ and save the resulting .ppk file in a safe location, but don’t close the puttygen window just yet. If you use multiple computers, putty will let you re-use your private key file between Windows machines, if that’s what you’re into. SSH on Linux may, but it will not let you use a puttygen file in a Linux system. (Based on that one time I tried it and it didn’t work for me.) So just keep that in mind.
Also, it’s no big deal to have multiple private/public key pairs on the same server. You can use a different pair for each client computer, which is probably safer and more convenient than using a shared key pair. If you lose access to a client machine for whatever reason, you can just delete the public key off of the server and that machine won’t be able to connect to your server.
Leave your puttygen window up and switch back to your putty/SSH window. Create a .ssh folder and a key file for SSH, then a text file to store your keys:
mkdir ~/.ssh
nano ~/.ssh/authorized_keys
Paste the Public Key text in the top of the puttygen window onto a single line in the file. This will be a Very Large Line Of Text(tm) (VLLOT). The VLOTT should begin with ‘ssh-rsa’ and end with ‘rsa-key-yyyymmdd’ where yyyymmdd is the date you created the key. Sometimes the key comment (steve@stevesblog.com in the example below) is the last bit of text. I haven’t quite nailed down why that is, presumably an order of operations thing. Anyway, be sure that the VLOTT begins with ssh-rsa, or you didn’t grab all the text in the public key.
Save and close the file (‘ctrl+x’ and then ‘enter’) and then set the permissions for the file:
chmod 600 ~/.ssh/authorized_keys
UPDATE: on CentOS 7 the home directory (~), the .ssh directory and the authorized_keys file should all be writable only by the owner so do this instead:
chmod 0700 ~
chmod 0700 ~/.ssh
chmod 0700 ~/.ssh/authorized_keys
Now exit your ssh session and reopen putty. You need to set the IP address of your server as the hostname. I prefer this to host names because DNS can’t always be trusted. Give your session a useful name.
Under ‘Connection -> Data’ add the username for your non-root account. In this example, I named my account ‘steve’.
Under ‘Connection -> SSH -> Auth’ browse to the safe place you saved your private key. You pasted your public key onto the server, and you have your private key stored on your computer. You will want to keep the private key file safe because if you lose it you have to set up a new pair while logged in at the console, which is a total pain. I keep mine in Dropbox so I can use them on multiple PCs, but I keep them secured with a passphrase.**
Now go back to Session and save your session profile. Henceforth you can connect simply by double clicking ‘steve’s server’ under ‘Saved Sessions’.
Now it’s time to test your new key pair. Just double click ‘steve’s server’ and you should be prompted for the passphrase that you set for your private key. Once you enter it, you should be logged in to the server as user ‘steve’. If you were able to log in using your key, you are all set to move on. You are now free to close PuttyGen.
* Protip: put your putty.exe file in ‘c:\windows\system32’ so you can run putty from the command line or the run line. If you want to be a real hard rock, rename putty.exe to ssh.exe. Did you know putty accepts commandline args? It does, so you can do awesome Unixy shit from the command line like type ‘ssh steve@testbox.stevesblog.com’ to connect to a remote host. It still pops up your connection in the putty window, but it keeps your hands on the keyboard. 🙂
** Another Protip: not setting a passphrase is handy for automating ssh connections, especially if you want to move files back and forth with ‘scp’ or mess with tunneling via local and remote ports. I haven’t found a decent scp command line app for Windows, other than the Unix utils in CygWin.
If The Server Rejects Your Key
It’s most likely that you didn’t paste the public key correctly. This is why we left the PuttyGen window open. 🙂
Log in with your non-root username and password (‘steve’ in this example) and open your ~/.ssh/authorized_keys file in nano again:
nano ~/.ssh/authorized_keys
In the PuttyGen window, make sure that you scroll to the top of the public key text. It should begin with ‘ssh-rsa’. Now click and drag down to the end of the public key text, then right click and select ‘copy’.
In the Putty window, with your authorized_keys file open in nano, delete the incomplete key and paste the complete text of the public key on a single VLLOT.
Save and exit nano, then exit your SSH session and try again.
Also make sure that you changed the permissions of the authorized_keys file:
chmod 600 ~/.ssh/authorized_keys
If your key is still being rejected, generate a new public and private key by clicking the ‘Generate’ button and starting the whole key process over again.
Disable Root and Cleartext Logins
Once your keypair is working, (and you are able to log in with it) it’s time to eliminate root logins and cleartext logins. Some folks will tell you that root logins are fine with SSH because passwords don’t get sent in the clear. While that’s true, ‘root’ is still the one username that is guaranteed to be on every Unix-based machine, so if you are going to brute force an account, this is the one to focus your efforts on. Disabling root logins and clear text logins is all done in the sshd_config file:
sudo nano /etc/ssh/sshd_config
Press ‘ctrl+w’ and search for the word ‘root’. You are looking for this entry:
# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
Change ‘#PermitRootLogin yes’ to ‘PermitRootLogin no’. (uncomment if necessary and change from ‘yes’ to ‘no’.)
Then press ‘ctrl+w’ and search for the words ‘clear text’. You are looking for this entry:
# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
Change ‘#PasswordAuthentication yes’ to ‘PasswordAuthentication no’ (uncomment and change from ‘yes’ to ‘no’.)
Once these changes are made, DO NOT LOG OFF OF YOUR SSH SESSION. Once these changes are implemented, it will be hard to log back in to undo anything if you make a mistake. You should have tested and succeeded with your ssh-key based login because we are about to restart the ssh daemon and prevent clear text logins:
sudo systemctl restart ssh
UPDATE: on Debian 9 you can restart SSH with
service ssh restart
UPDATE: on CentOS 7 you can restart SSH with
systemctl restart sshd.service
To test ssh logins, connect to the IP of your server with putty using the ‘Default Settings’ profile. Your login attempt should fail because only people with private keys are allowed to the party:
At this point you are far from being hack-proof, but you are a bit more locked down than you were before, and there are always more convenient targets out there 🙂
Hardening web servers is another story, which really isn’t my bag to be honest. There’s a reason that I host my blogs with Google or WordPress 🙂
OH SHIT! I set this up on my Windows machine at home and I don’t have access to my private key at work/school/Aunt Tillie’s computer!
So you’ve set up your keys, you’ve disabled clear text logins and now you are trying to get a new public key onto your locked down box, but you can’t log in because you don’t have the private key. How screwed are you?
Not completely.
I find myself in this situation when I am setting up a new Linux or Unix workstation. With Windows I use a cloud storage service to keep my keys (that have passphrases), and I use one key pair per server, and just reuse my private key on each of my Windows machines.. The passphrase protects the private key on the cloud service (should the cloud service experience a breach or some other security failure) and on the local drive of my PC, should it fall into the wrong hands.
In the Unix world, I do the opposite. As I stated above, SSH keys are way easier to do with Unix. It’s no problem to produce key pairs and upload them to remote servers, so I use one key pair on per workstation, but then I use that pair on each of my servers.
This is probably a much safer practice than my Windows+Dropbox approach, but I use encryption tools like BitLocker and KeePass to add an little more security when I use Windows.
All you have to do on your new Unix box is create a new key pair like so:
ssh-keygen -t rsa -b 4096
And then use the ever so handy tool ssh-copy-id to add the newly created public key:
ssh-copy-id -i ~/.ssh/id_rsa user@remote-server
It couldn’t be easier, except when you have disabled password logons like an absolute walnut. In that case, you will need to log into both your new Unix box and your locked down remote Unix box from a computer that already has a key pair configured with the locked down remote server. For this reason, I recommend setting up a VM, a container, or a shell account to use as an intermediary.
By default, RSA key pairs will be stored in ~/.ssh/id_rsa (private) and ~/.ssh/id_rsa.pub (public). Once you are logged in on your third computer that is already set up with a key pair, open an SSH connection to the new Unix machine and an SSH connection to the locked down remote server. For the purposes of this demonstration, I will call the new box “newbox” and the locked down remote server “remotebox”. Both non-root user accounts will be called “steve”.
In the window for your session on newbox, view your id_rsa.pub file like so:
steve@newbox:~$ cat ~/.ssh/id_rsa.pub
You will see the familiar Very Long Line of Text ™ (VLLOT) which is the public key. The text will fill multiple lines. This is important to note, because you will need to copy all of these lines in a moment.
In the window for your session on remotebox, edit your authorized_keys file like so:
steve@remotebox:~$ nano ~/.ssh/authorized_keys
You will see the VLLOT that you pasted in when you locked down the server initially. It will be all on one line, and will most likely be truncated. This is important to note because you will be pasting text into this window in a moment.
Now you just copy the VLLOT from the window with your session on newbox into the editor window with your session on remotebox. If you are using putty for this operation, you can copy text by pressing ALT+C, instead of CTRL+C. CTRL+C cancels things in the Unix shell, and in Nano it will show the current cursor position.
Your window with our session on remotebox should now have two Very Long Lines Of Text(tm). Use the arrow keys on your keyboard to verify that the whole line is there. If you are certain that the VLLOT was pasted correctly, simply save the authorized_keys file and exit. In Nano, CTRL+O to saves and CTRL+X exits.
Once you have saved the authorized_keys file, switch to your window with your session on newbox. You can now attempt to connect to remotebox via SSH:
steve@newbox:~$ ssh steve@remotebox
If all goes well, you should be prompted for your passphrase for “/home/steve/.ssh/id_rsa”. If not, you probably didn’t paste the VLLOT correctly.
Public Access Unix Rocks
As I stated before, I recommend having a VM, container, or a Unix shell account to use as an intermediary for accessing locked down remote servers. Getting remote access to your gear is important. So important that I run multiple VMs to make sure that I can access everything remotely.
If you are not fortunate enough to be blessed with an embarrassment of hardware like I am, you can still use a dedicated server as an intermediary by signing up for a shell account with SDF.
I have had a shell account at SDF for decades. It enabled me to learn about large Unix systems without needing to set up a Linux box. Even if you have a server at home and a hosted server somewhere else, having a Unix shell account is still a great tool to have in your arsenal for way more than just stupid SSH tricks.
I am a huge fan of screen. It’s indispensable for working on a Unix host via SSH. It lets me have multiple terminals (screens) up at a time. There are dudes that use screen to split their terminals into multiple views, like a tiling window manager, but for the command line.
My needs are not nearly as sophisticated, since I mostly use putty to connect to Linux servers from Windows.
I use 4 special keys:
F9 to detach from the screen session. This is leaves your session running in the background. I mostly use this to idle in IRC. Once detached from your session you can view your active screen session by typing:
screen -ls
Which will return something like this:
user@localhost:~$ screen -ls
There is a screen on:
2030.pts-0.localhost (05/25/2016 06:45:51 PM) (Detached)
1 Socket in /var/run/screen/S-user.
To reconnect to a detached screen session, type
screen -r 2030.pts-0.localhost
If the session is in use elsewhere, use the -D option:
screen -D 2030.pts-0.localhost
This will disconnect the screen session that’s in use, log off the SSH session that initiated it, and then reattach the active SSH session to the screen session.
Or, if you’re like me and never have any idea if you have a running a screen or not, just combine -D and -R and quit worrying about sockets and get on with your life:
screen -DR
And, if you are also like me and forget the switches for screen, just use the alias command in your .bashrc to have screen do -DR every time:
alias screen = 'screen -DR'
F10 to open a new terminal in screen.
This option lets you have multiple terminals in the same SSH session. This is handy for having a full screen app (like irssi) in one term, and one or more additional terms for running other commands. To close a terminal, type
exit
F11 and F12 to switch terminals
When you have multiple terminals open you can navigate them, from left to right with the F11 key to select the terminal to the right, and the F12 key to select the terminal to the left.
The File
To use this file, simply paste the contents below into a file called .screenrc in your home directory. So here it is, the .screenrc, that I have been using for years:
startup_message off
# Window list at the bottom.
# I got the long line of vars from https://bbs.archlinux.org/viewtopic.php?pid=423481#p423481
hardstatus alwayslastline
hardstatus string "%{.kW}%-w%{.W}%n %t%{-}%{=b kw}%?%+w%? %=%c %d/%m/%Y" #B&W & date&time
# From Stephen Shirley
# Don't block command output if the terminal stops responding
# (like if the ssh connection times out for example).
nonblock on
# Allow editors etc. to restore display on exit
# rather than leaving existing text in place
altscreen on
# bind F9 to detach screen session (to background)
bindkey -k k9 detach
# bind F10 to create a new screen
bindkey -k k; screen
# Bind F11 and F12 (NOT F1 and F2) to previous and next screen window
bindkey -k F1 prev
bindkey -k F2 next
chrizzle23.com 
You must be logged in to post a comment.