Adventures in Proxmox Part 3: Chris don’t know shit about networking

When I first started messing with Proxmox, I crashed my home network. If you aren’t interested in the story of my journey of network sexual awakening, click here.

I have since spent the last several months learning about Proxmox networking using virtual box. I have also been working on a parallel project: upgrading my home network to be segregated using VLANs. Like my budget for server hardware, my budget for network gear is practically nonexistent, so I have been doing a lot of reusing things that should have been replaced years ago.

After a bit of consternation, I settled on a prosumer router and a smart switch, rather than a PC-based router and a managed switch. Mostly because I needed this to work for the family as well as for the lab, and I didn’t want to spend weeks relearning Cisco. Time to tear down the old home network!!

A New Router

My plan is to have 4 “real” networks for my “physical” equipment:

1. The family’s wireless network – for phones, tablets, game consoles, and tv sticks.
2. My wired network for my personal workstations and servers.
3. A VOIP network for POE phones, ATAs, and my PBX.
4. A server and network lab for me to wreck things.

When I say “real” I really mean “operated by humans” or perhaps “not a Proxmox host”. When I say “physical” I also mean “operated by humans” or perhaps “not a Proxmox host”. At least half of these “real” ports are VLANs, and at least half of these “physical” devices are VMs. In this scenario, “real” and “physical” networks and devices are the ones that I and the family use, compared to the networks that are dedicated to the Proxmox cluster.

The critical distinction is that all of these network segments connect to a different port on the router, and have firewall rules to keep them from connecting to each other. In this scenario, a dumb switch plugged into each port of the router will provide a physically separated network at layer 2 (Ethernet) and a logically separated network at layer 3 (IP). It is here that I have used my first batch of dumb old mini switches:

1. eth1 – Family Wireless, 192.168.10.0/24
2. eth2 – Personal Wired, 192.168.11.0/24
3. eth3 – VOIP, 192.168.12.0/24
4. eth4 – Lab, 192.168.13.0/24

The family wireless network consists of 2 wireless access points, both with 4 dumb gigabit Ethernet ports:

1. WAP port 1 -> eth1 on the router, uplink to the Internet
2. WAP port 2 -> eth0 on the NAS appliance
3. WAP port 3 -> port 1 on the smart switch
4. WAP port 4 -> port 1 on the other WAP

So, I had my router set up, and plugging a laptop in to each dumb switch let me pull an IP from the DHCP server for the respective network segment. I was also able to browse the Internet. Awesome. I have managed to convert a big, clunky, error-prone network into four smaller error-prone networks. This is progress?

As far as the family is concerned, eth1 on the router is the network. Wireless access to both the Internet and to the data and media stored on the NAS. If I never plug in the smart switch then only I would notice. I have the WAP’s dumb switch plugged in to the smart switch because I have a media server VM on the Proxmox cluster that I want to put onto the wireless network to stream video to tablets, mobile phones and smart TVs. Because the cluster nodes only have 4 network ports, I need to put multiple network connections on to 1 of those network ports. This is where VLANs come into play. This is also where upgrading my knowledge of routing, switching, and firewalls comes in to play with Proxmox: putting the cluster onto all 4 of my network segments using just one network port from each node.

VLANs: everything you hate about dozens of dumb switches, plus virtualization

With the new router working, it’s time to configure the networks’ core: the smart switch.

VLANs are a great way to divide up a big physical switch into smaller virtual networks. A 24 port switch could be broken down into 4 networks, with a a varying number of ports in each network. You can also put a single switch port onto more than one VLAN. The network traffic gets put into the appropriate virtual network by using tags. You can even put a given port into “all” of the VLANs, this is sometimes referred to as a “trunk.” Trunks are used to connect multiple switches together, passing all tags between them.

Dumb switches can’t tag traffic. So, if you want to mix a smart switch that does VLANs with a dumb switch that doesn’t, you need to make sure that your untagged traffic is going out of the right ports. In the hypothetical 24 port managed switch in the example above, if you put port 2 into VLAN 2, and then plug a dumb switch into port 2, then port 2 needs to know what to do with untagged traffic. Traffic coming out of the dumb switch won’t have tags, and traffic going into to the smart switch will lose its tags. This is the essence of “VID” and “PID/PVID”. A VID is a VLAN ID, PVID is a Port VLAN ID. All the ports on the smart switch need to treat all traffic as tagged, even when it’s not. Untagged traffic needs to be treated differently than tagged traffic, basically meaning that “untagged” is just a special category of “tagged”. The PVID is a kind of “untagged == special tag” way for ports to deal with untagged traffic. This is the exact moment that I developed a migraine.

I have done a decent job keeping the family wireless network packets away from everything, and everything away from the family by putting each network segment on its own dumb switch. Now it is time to blur those boundaries a bit by plugging each of those dumb switches into the smart switch. My network is broken into 4 subnets, so my VLANs will break down something like this:

• VLAN 1 – Family Wireless
• VLAN 2 – Personal Wired
• VLAN 3 – VOIP
• VLAN 4 – Lab

I probably don’t need a separate /24 (class C) network for each VLAN, but I am not very clever and I have zero confidence in my ability to design networks or IP schemes. I know how routing works when you are using /24’s so for my implementation VLAN == /24. Also, as I learned in the Virtual Box lab, network designs get real confusing real fast, so having the VLAN tag roughly correspond to /24 subnet helps me to not go completely insane.

The smart switch is configured by a web interface. This interface has a default IP of 192.168.0.1, so I set a static IP on the Ethernet port of my laptop and logged in. This part of the configuration is important, and it will come into play again later. Once I have all the VLANs set up, I still need to be able to access the switch on this IP address.

I configured the first 4 ports on the switch as access ports or up-links to the dumb switches. Because the dumb switches don’t tag traffic, I needed the uplink ports to treat all “untagged” traffic as tagged to a single VLAN, using the PVID:

• switch port 1 – VLAN 1, PVID 1
• switch port 2 – VLAN 2, PVID 2
• switch port 3 – VLAN 3, PVID 3
• switch port 4 – VLAN 4, PVID 4

So now, if I change port 5 to VLAN 1 and PVID 1, I can plug in my Windows laptop and pull an IP from the wireless network. Then I can change port 5 to VLAN 2 and PVID 2, and now I can pull an IP from the wired network. Now I need to figure out how to get my Prox cluster nodes to sit on all 4 networks at the same time using a single switch port for each node.

Enter the Management Workstation

Up to this point, I was able to set up my dumb switches and my VLANs with a Windows laptop. I just disabled the WiFi and plugged the Ethernet adapter into the various switches and ports. This was fine for scenarios where one switch port corresponded to just one network segment. But it turns out that Windows can’t do VLANs without proper hardware and software support for the NIC. If you have a VLAN-aware NIC and the Intel or HP enterprise app to configure it, I guess it works fine, but there is no Windows 10 app for the Intel NIC in my crashtop.

In my Virtual Box Proxmox lab, I learned that life is just easier when you have a Linux box dedicated to managing the cluster and testing your network setup, so I decided that before I set up the cluster, I should set up a “Management Workstation.” For the BoxProx lab, I used a Virtual Box VM running a GUI to administer the cluster because I needed a browser on the host only network. Technically, I could have run the management workstation without a GUI and just used SSH tunneling to access the web management interfaces for the Proxmox VMs, but I didn’t want to spend any time doing stupid SSH tricks. I also don’t have the actual hardware cluster running yet, so I need to do this with actual hardware. The hope is that once I get the VLANS and network bridges configured, the workstation will be superfluous. Therefore, the workstation doesn’t have to be powerful at all. Literally any old laptop or desktop that is laying around will do nicely.

My operating system of choice is Turnkey Linux Core. Set up an old desktop on port 5 of the smart switch. For the initial install, I left port 5 configured for VLAN 1 and PVID 1. I was able to pull an IP address from the wireless network, install and update the OS, and configure SSH.

Remote access is important because I can’t sit in my basement all day; Internet access is important because I need to install some network tools.

First step is to get the VLAN tools installed:

apt-get install vlan

Then enable VLAN support in the kernel:

echo 8021q | tee -a /etc/modules

Then add your tagged network interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

Then reboot the machine. I know there is a bunch of crap that you can do to avoid that, but this is the only way I can be sure that it works. I also know that if you name the interface eth0.N you probably don't have to mark the 'vlan-raw-device' but the Debian VLAN tutorial did it so I did it too.

What this does is change the IP of untagged interface eth0 to 192.168.0.10 (remember the IP of the switch from before?) and add eth0.1 (VLAN 1) with an IP of 192.168.1.10 and configured a default gateway and DNS for that interface.

Now, the machine should still be connected to the Internet, and you can modify port 5 on the smart switch to be in VLAN 1 and PVID 1.

If you can ping the IP for the smart switch (192.168.0.1), the IP of something on your wireless network (like an access point) as well as Google's DNS (8.8.8.8) then you are in good shape.

At this point, I left the basement and went upstairs. I connected my laptop to the family wireless network (192.168.1.0/24) to SSH into the workstation. Since I will be making modifications to the smart switch configuration, as well as the management workstation, I decided to configure PuTTy to drop a local port and forward it to 192.168.0.1:80 so that I can access the web interface of the smart switch from my laptop, and the unencrypted HTTP traffic will be secured by the SSH tunnel.

Now I just need to move the Internet access to the 'Lab" VLAN and add the remaining VLANS to /etc/network/interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0

auto eth0.2
iface eth0.2 inet static
    vlan-raw-device eth0
    address 192.168.2.5
    netmask 255.255.255.0

auto eth0.3
iface eth0.3 inet static
    vlan-raw-device eth0
    address 192.168.3.5
    netmask 255.255.255.0

auto eth0.4
iface eth0.4 inet static
    vlan-raw-device eth0
    address 192.168.4.5
    netmask 255.255.255.0
    gateway 192.168.4.1
    dns-nameservers 8.8.8.8 8.8.4.4

The last step is to make sure that smart switch port 5 is part of VLANs 1, 2, 3, and 4, with PVID 1. If all goes well, the workstation can ping the smart switch IP, Google DNS, and servers on all 4 VLANs.

The next step is to use this same network setup for the management NIC on the Proxmox cluster. Using the 4 VLAN interfaces for the network bridges (VMBR1-VMBR4).

Stupid SSH Tricks

I use this site for a number of reasons. One them is to keep a little diary of things that I have figured out so that I can reference them later. The problem is that those little discoveries are buried in rambling posts about why I chose to do something. So here is where I talk about setting up a dynamic port to use as a SOCKS proxy for securing web traffic, local ports for accessing resources inside of closed network, and remote ports for making resources inside of a closed network publicly available on the Internet.

I have given 2600 talks about SSH tunnels but I really don’t have a permanent record of my various [mis]uses of SSH, so I thought I would put them all in a post for future reference. I have written about securing SSH with asymmetric encryption keys, but there are many more things that you can do with SSH.

IMPORTANT NOTE: – none of these tricks work unless your host has TCP port forwarding enabled. This option is usually enabled by default, but you should double check your /etc/ssh/sshd_config file:

nano /etc/ssh/sshd_config

Then make sure that the option is uncommented and enabled:

 AllowTcpForwarding yes

Also, if all this putty crap looks tough to do, just remember that all of this is way easier with UNIX. Particularly using the -J option for jumping SSH connections.

Dynamic Ports: slip past content filters with this one weird trick

This is the go-to use for an SSH tunnel. University and corporate networks are often overseen by petty tyrants who limit access to sites of questionable content, or sites that are prone to eating bandwidth, like social and streaming media. Ironically, most of these networks allow outbound SSH. SSH kills content filters dead.

You can create a SOCKS proxy using an SSH connection by creating a dynamic port. You can then point your browser to use your local address (127.0.0.1) as a SOCKS proxy to smuggle all of your browser traffic thru the SSH tunnel.

I like using Firefox specifically for this task because Chrome or IE does dumb things when you mess with the proxy settings. Or rather, they did dumb things that one time I used them to tunnel 10 years ago and then decided on using FireFox. If you want to also tunnel your DNS queries, in Firefox type “about:config” in the browser bar and then find the setting “network.proxy.socks_remote_dns” and set it to “true”. There’s probably a more modern way to to that, but that’s the tried and true way. Tunneling DNS isn’t strictly necessary, but it does help you stay under the radar on a restrictive network.

It should be noted that once you tunnel your browser traffic through the tunnel, you are likely to lose access to intranet sites on your local subnet. For this and a number of other reasons, I like to run Brave (Chrome) for my normal stuff, and Firefox for the tunneled stuff. Running two browsers at the same time seems wasteful, but it saves a bunch of headaches.

On Windows, you can drop the port one of three ways:

If you are into radio buttons and text boxes, you can configure PuTTY to open one every time you connect. On the tree control to the left, click Connection -> SSH -> Tunnels. You’ll need to select dynamic and enter a source port (5555 in the picture to the right) and click ‘Add’. Your new port should appear in the list above as ‘D5555’. Be sure to go back to your session and click ‘Save’ of you want the port to be created every time you open the session. As long as you are messing with your PuTTY session, you might as well set your terminal colors to green text so you look like a real hacker.

If using the GUI makes you feel like a wimp, you can just script it with a handy batch file:

putty -D 5555 user@ssh.server.com

You can also feel like a real UNIX badass by copying putty.exe to c:\Windows\System32 and renaming it to ssh.exe so you can kick off your session from DOS like a real console cowboy.

OR you can dump the whole interactive shell pretense and use plink.exe to make your connection and drop your ports without the whole pesky PuTTY window getting in your way:

plink -D 5555 user@ssh.server.com

Plink is bascially PuTTY with no window. It functions basically the same in all other respects.

If you are using a Unix or Linux workstation, you can set up your dynamic port with a similar syntax:

ssh -C -D 5555 user@ssh.server.com

Note: the -C switch compresses traffic going thru the tunnel, theoretically increasing network speeds through the tunnel.

Local Ports: I heard u like shells so we tunneled SSH thru yo SSH so you can get a shell while you gettin a shell

You can use SSH to secure more than just your web browsing. You can use it to secure pretty much any TCP connection. I like using it to secure notoriously insecure VNC and X sessions.

Another use is to get around port restrictions. Some networks may allow outbound SSH, but only on port 22. Some home ISP’s get shitty about running servers on reserved ports like 22. This means you have to forward some bunk port on your home router, like 62222 to the SSH server on your home network. I used to do this when I had Time Warner cable. The problem would get worse when I was trying to connect remotely from a restrictive network that only let SSH out on port 22.

To get around this problem, I would have to SSH to a public access Unix system like the Super Dimensional Fortress on port 22 and then drop a local port that forwarded to the bunk SSH port on the IP of my home router. When I did that with different windows that had different text colors it looked like I was on CSI: Hacker Girl back tracing the killer’s IP address.

The setup is pretty much the same as for the dynamic port, only you have to specify a destination IP and port as well. The idea is to connect to a remote host that is on the other side of a restrictive firewall and using the SSH tunnel to make something accessible to your local your local network. It forwards all traffic to your local port to the remote destination thru the tunnel. In the example above it was SSH but it could be RDP or any other TCP connection. I’ll use RDP (port 3389) in the example below.

To tunnel a Microsoft Remote Desktop session through SSH using the PuTTY gui, use the tree control to the left, click Connection -> SSH -> Tunnels. You’ll need to select local and enter a source port (13389 in the picture to the right), set the destination address or host+domain name and the port (192.168.1.10:3389 in the picture to the right) and click ‘Add’. 13389 will be the port on your workstation that is now connected to the RDP port on the remote network (3389). Your new port should appear in the list above as ‘L13389 192.168.1.10:3389’. Be sure to go back to your session and click ‘Save’ of you want the port to be created every time you open the session. In your RDP client, you would connect to 127.0.0.1:13389.

If you are scripting this setup, use the -L switch along with your source port, destination IP/host and the destination port. Using the scenario from above, you forward local port 13389 to 192.168.1.10 port 3389 like this, where your SSH username is ‘alice’ and your home network’s dynamic DNS hostname is casa-alice.dynamic.DNS:

putty -L 13389:192.168.1.10:3389 alice@casa-alice.dynamic.DNS

And finally, the syntax is the same with plink:

plink -L 13389:192.168.1.10:3389 alice@casa-alice.dynamic.DNS

You can actually specify multiple local ports to remote destinations. I do this with PuTTY to get direct access to the web interface on my Proxmox cluster and to RDP to a Windows host using just the one tunnel and without having to mess with my proxy settings.

Remote Ports: SSH kills firewalls dead.

In the local port scenario, you are connecting to a remote host behind a firewall and using the SSH tunnel to make a host inside the remote firewall accessible to your local your local network. You can also do the opposite, which is to connect to a remote host outside of your firewall and use the SSH tunnel to make a host inside your local firewall accessible to either the Internet or to hosts on a remote network.

You do this by dropping a port at the other end of the tunnel. On the remote host. The obvious use is to temporarily punch a hole in the local firewall to expose an internal web server to the Internet. If the remote host that you are connecting to is directly connected to the Internet, (like a hosted VM from Cloud At Cost) you can temporarily open a port on the remote server to tunnel traffic to the web server on your internal network.

A more nefarious use for a remote port would be for a leave-behind box (formerly known as a dropbox before the term became a brand name for cloud storage) to phone home from a target network. Basically you build a cheap single board PC, like a Raspberry Pi or a plug server that you physically plug into a network that you plan on hacking testing for security holes. This approach saves a ton of time reverse engineering firewalls to gain access. There are two basic approaches, load up the box with tools and drop it, or use a minimal box as a router/pivot for tools you are running outside the target network.

To do this with the PuTTY gui, it’s basically the same as setting up a local port. Use the tree control to the left, click Connection -> SSH -> Tunnels. You’ll need to select remote and enter a source port (58080 in the picture to the right), set the destination address or host+domain name and the port (192.168.1.10:80 in the picture to the right) and click ‘Add’. You also need to click the check both of the boxes next to “Local ports accept connections from other hosts” and “Remote ports do the same (SSH-2 only)”. Your new port should appear in the list above as ‘R58080:192.168.1.10:80’. Be sure to go back to your session and click ‘Save’ of you want the port to be created every time you open the session.

If you are scripting this setup, use the -R switch along with your source port, destination IP/host and the destination port. Using the scenario from above, you forward remote port 58080 on ssh.server.com to port 80 on your internal home web server with the IP 192.168.1.10 like this:

putty -R 58080:192.168.1.10:80 alice@ssh.server.com

And finally, the syntax is the same with plink:

putty -R 58080:192.168.1.10:80 alice@ssh.server.com

The only gotcha with scripting your remote port drop with putty/plink is that I don’t think that there is a command line switch for enabling connections from other hosts, so you have to enable the remote port sharing on the SSH server side.

Making Local and Remote Ports Accessible to other hosts

Sharing your remote and local hosts lets you set up your SSH tunnel on one host and then connect to the tunnel from a different host.

In the case of a local port, you could initiate the SSH session on your home Linux server, and then connect to that port from your Windows workstation. This is handy if you are tunneling RDP and you don’t have PuTTY available on your Windows box. Although PuTTY is super portable so it’s dead simple to smuggle it onto the most locked down of Windows machines.

In the case of the remote port, it’s pretty much mandatory for the web server or dropped box use cases. You can still script the connection, you just have to modify your sshd_config file on your SSH server. On a Debian-esque server you do this by using sudo or su to become root and then type:

nano /etc/ssh/sshd_config

You then add the GatewayPorts option. You can put it anywhere in the file, but I prefer to keep it in the first few lines of the file where entries for port configuration are.

# What ports, IPs and protocols we listen for
Port 22
GatewayPorts yes

Then save the file and restart SSH:

systemctl restart ssh

Or on Debian 9, you use ‘service’ to restart SSH:

service ssh restart

I am sure that this option is a big security risk, so I recommend a cheap low powered VM dedicated specifically to bouncing SSH connections. I also recommend securing it with SSH keys. If you are looking to script SSH connections that are secured with SSH keys, I recommend not setting a passphrase on your private key. You can include your private key on the putty/plink commandline with the -i switch:

putty -L 13389:192.168.1.10:3389 alice@casa-alice.dynamic.DNS -i c:\path\to\your\key\key.ppk

My guide to setting up SSH keys with Putty

TL;DR: if you just want to set up keys with putty: IDGAF about Cloud At Cost take me to the Putty screenshots. If you are sitting in front of a computer that you don’t have keys set up on, and you are trying to log into a remote server that you have already locked down: I haven’t set up keys on my other computer like an absolute walnut.

Fun with Cloud At Cost
I have become a kind of fan of Cloud At Cost. Their one-time-fee servers and easy build process is great for spinning up test machines. I would hardly recommend running anything that I would consider “production” or mission critical on a cloud at cost VM, but it is a cheap, quick, and simple way to spin up boxes to play with until you are ready for more expensive/permanent hosting (like with Digital Ocean or Amazon). Spinning up a new box means securing SSH. So here is my guide.

The major problem with a hosted server of any kind is drive-by scans. There are folks out there that scan for huge swaths of the Internet looking for vulnerable machines. There are two basic varieties: scanning a single host for all vulnerabilities, and scanning a large number of hosts for a specific vulnerability. A plain box should really only be running SSH, so that is the security focus of this post. There should also be a firewall running, that rejects connections on all ports except the services you absolutely need, but that’s a blog post for another day.

It should be noted that Your security measures don’t necessarily have to be top notch, your box just has to be less convenient than the next host on the scanners’ lists. It’s not hard to scan a large subnet and find hosts to hammer on. Drive-by scans are a numbers game; it’s all about the low hanging fruit. With C@C, it’s a question of timing. You have to get onto the box and lock it down quickly. Maybe I’m just being paranoid, but I have had boxes that I didn’t log in to right after spinning them up and I have seen very high CPU utilization on them when they aren’t really running anything, which leads me to believe that the host has been compromised. Also, beware that the web-based stats can be wildly inaccurate.

This guide will only lock down SSH. If you are running a web server, this guide will not lock down the web server. If you are running Asterisk, this guide will not lock down Asterisk. If you are running MySQL, for the love of god make sure that it’s only accessible from localhost (127.0.0.1) and that is not accessible from the Internet. All this guide will do is shore up a couple of vulnerabilities with SSH. I recommend running these steps *BEFORE* installing anything on your VM.

My use case for Cloud At Cost is something like this: There are times when I need a box that is easier to get to than hosting a box on my home network, but doesn’t really justify the monthly cost of running a server on Linode, Digital Ocean or Amazon. For me, I spend a lot of time working all night inside a very restrictive corporate network, so it’s hard to get access to my stuff at home especially since Team Viewer is compromised. C@C is cheap and easy, which probably means it’s a playground for scammers and other bad actors. This means it’s a good idea to lock down your box before you do anything useful with it, and keep the useful things that you do with it to the bare minimum.

You can get started with C@C for around $35, but if you follow them closely, you can catch some of their discount deals and get a very low end developer box for around $10. I took advantage of a few of these promotions and now I have a bucket of resources at my disposal for all of my tinkering needs. Also, if your box starts to misbehave (loads of network traffic, high cpu utilization, etc.) it’s probably compromised, so just torch it and build a new one.

Getting Started

You can learn about the basics of the Cloud At Cost panel here, the info will be useful later on:

Once you have signed up with C@C, bought some resources, and fired up your Linux VM, it’s time to do some housekeeping. I prefer Debian, and it’s what I am using in this guide, but it doesn’t really matter what you choose.

As soon as the box is up, log in with SSH, using the root password given in the information button. I use putty*, because most of my time in front of a computer is spent working or gaming, so I use Windows a lot. I know it upsets a lot of folks to hear that, but hey, those folks can feel secure in knowing that their “Unix Beards” are mightier than mine.

The very first thing that I do is change the root password. Make like 30 or more random characters. You shouldn’t actually need to type it in after this point, but keep it somewhere encrypted just in case. I also comment out the non-us repo that C@C Debian machines are still pointed to in sources list:

passwd
nano /etc/apt/sources.list

Just locate the line that begins with “deb http://non-us.debian.org” and put a # in front of it. On a C@C Debain 8 box, it should be the first line.

With that pesky non-US entry removed, you are clear to update your packages:
apt-get update
apt-get upgrade

I also run these commands from the Nerd Vittles blog to make sure the password doesn’t revert to the Cloud At Cost root password:

sed -i '/exit 0/d' /etc/rc.local
killall plymouthd
echo killall plymouthd >> /etc/rc.local
rm -f /etc/rc3.d/S97*
echo "exit 0" >> /etc/rc.local

I don’t know if they are strictly necessary, but the dudes at Nerd Vittles recommend it, and they spend waaaay more time doing this stuff than I do, so there you have it.

After that, it’s time to install fail2ban, and then create a non-root user:

apt-get install fail2ban
adduser steve

Hopefully, in a few minutes fail2ban will be made superfluous by our additional security measures. In the meantime it will stop brute force attempts. Some of my hacker buddies change the default port for SSH to throw off driveby scans, but the restrictive corporate network I mentioned before doesn’t like arbitrary ports, so that’s a hard no in this case.

Enable Sudo for a Non-Root User

To start implementing our security measures, we will install sudo, add ‘steve’ (our non-root user) to the sudo group, and then make sure steve has the right permissions in the sudoers file:
apt-get install sudo
adduser steve sudo
nano /etc/sudoers

At this point the /etc/sudoers file should open in the Nano next editor. I know I should be using vi, but I am too busy #YOLOing to do that Unix Beard crap. 🙂

Press ‘ctrl+w’ to open the search box, and type ‘%sudo’ to find the permissions line.
Press ‘ctrl+k’ to cut the ‘%sudo ALL=(ALL:ALL) ALL’ line, and then ‘ctrl+u, ctrl+u’ (hold ctrl and press ‘u’ twice) to paste the line in twice.
Edit the second line to read ‘steve ALL=(ALL:ALL) ALL’ and press ‘ctrl+x’ to exit, and press enter to save.

Setting up sudo is important because we are going to disable root logins here in a minute, but first we are going to set up SSH Keys for logins and then disable clear text logins. SSH does use clear text passwords, but it passes them through an encrypted tunnel. This means that while your password isn’t likely to be sniffed, it could be guessed or brute forced. Using SSH keys means you have to have the right private key to match with a public key on the server. But before we can do any of that, we need to test the new non-root account by logging in with it.

Once you are logged in as steve, test sudo:
sudo whoami

Which should return ‘root’.

Securing SSH with Asymmetric Keys

Once the non-root account is working and sudo-ing, we can proceed to lock down SSH with public+private key pairs. I will explain how to do this with putty for Windows, but it’s actually way easier to do this with Unix.

The first step is to make sure you have puttygen.exe handy. Download it and launch it, change the bits for your keys to 4096 (in the lower right corner) then click the ‘Generate’ button.

Wiggle the mouse around for a bit, and in a minute or so you will see your public key, with a key comment and blanks for your passphrase. You don’t have to change the comment, or enter a passphrase, but I recommend it. I like to change the comment to match the username and server (‘steve@stevesblog.com’ in the screenshot below), since I have lots of different keys. The passphrase keeps things safe in case your private key file falls into enemy hands.**

At this point, you may be tempted to use the same passphrase for your private key as you use for your non-root user account. This is a bad idea, because your non-root password is now basically your root password. Do yourself a favor and use two completely different passwords.

Next, click ‘Save private key’ and save the resulting .ppk file in a safe location, but don’t close the puttygen window just yet. If you use multiple computers, putty will let you re-use your private key file between Windows machines, if that’s what you’re into. SSH on Linux may, but it will not let you use a puttygen file in a Linux system. (Based on that one time I tried it and it didn’t work for me.) So just keep that in mind.

Also, it’s no big deal to have multiple private/public key pairs on the same server. You can use a different pair for each client computer, which is probably safer and more convenient than using a shared key pair. If you lose access to a client machine for whatever reason, you can just delete the public key off of the server and that machine won’t be able to connect to your server.

Leave your puttygen window up and switch back to your putty/SSH window. Create a .ssh folder and a key file for SSH, then a text file to store your keys:
mkdir ~/.ssh
nano ~/.ssh/authorized_keys

Paste the Public Key text in the top of the puttygen window onto a single line in the file. This will be a Very Large Line Of Text(tm) (VLLOT). The VLOTT should begin with ‘ssh-rsa’ and end with ‘rsa-key-yyyymmdd’ where yyyymmdd is the date you created the key. Sometimes the key comment (steve@stevesblog.com in the example below) is the last bit of text. I haven’t quite nailed down why that is, presumably an order of operations thing. Anyway, be sure that the VLOTT begins with ssh-rsa, or you didn’t grab all the text in the public key.

Save and close the file (‘ctrl+x’ and then ‘enter’) and then set the permissions for the file:
chmod 600 ~/.ssh/authorized_keys

UPDATE: on CentOS 7 the home directory (~), the .ssh directory and the authorized_keys file should all be writable only by the owner so do this instead:
chmod 0700 ~
chmod 0700 ~/.ssh
chmod 0700 ~/.ssh/authorized_keys

Now exit your ssh session and reopen putty. You need to set the IP address of your server as the hostname. I prefer this to host names because DNS can’t always be trusted. Give your session a useful name.

Under ‘Connection -> Data’ add the username for your non-root account. In this example, I named my account ‘steve’.

Under ‘Connection -> SSH -> Auth’ browse to the safe place you saved your private key. You pasted your public key onto the server, and you have your private key stored on your computer. You will want to keep the private key file safe because if you lose it you have to set up a new pair while logged in at the console, which is a total pain. I keep mine in Dropbox so I can use them on multiple PCs, but I keep them secured with a passphrase.**

Now go back to Session and save your session profile. Henceforth you can connect simply by double clicking ‘steve’s server’ under ‘Saved Sessions’.

Now it’s time to test your new key pair. Just double click ‘steve’s server’ and you should be prompted for the passphrase that you set for your private key. Once you enter it, you should be logged in to the server as user ‘steve’. If you were able to log in using your key, you are all set to move on. You are now free to close PuttyGen.

* Protip: put your putty.exe file in ‘c:\windows\system32’ so you can run putty from the command line or the run line. If you want to be a real hard rock, rename putty.exe to ssh.exe. Did you know putty accepts commandline args? It does, so you can do awesome Unixy shit from the command line like type ‘ssh steve@testbox.stevesblog.com’ to connect to a remote host. It still pops up your connection in the putty window, but it keeps your hands on the keyboard. 🙂

** Another Protip: not setting a passphrase is handy for automating ssh connections, especially if you want to move files back and forth with ‘scp’ or mess with tunneling via local and remote ports. I haven’t found a decent scp command line app for Windows, other than the Unix utils in CygWin.

If The Server Rejects Your Key

It’s most likely that you didn’t paste the public key correctly. This is why we left the PuttyGen window open. 🙂

Log in with your non-root username and password (‘steve’ in this example) and open your ~/.ssh/authorized_keys file in nano again:
nano ~/.ssh/authorized_keys

In the PuttyGen window, make sure that you scroll to the top of the public key text. It should begin with ‘ssh-rsa’. Now click and drag down to the end of the public key text, then right click and select ‘copy’.

In the Putty window, with your authorized_keys file open in nano, delete the incomplete key and paste the complete text of the public key on a single VLLOT.

Save and exit nano, then exit your SSH session and try again.

Also make sure that you changed the permissions of the authorized_keys file:
chmod 600 ~/.ssh/authorized_keys

If your key is still being rejected, generate a new public and private key by clicking the ‘Generate’ button and starting the whole key process over again.

Disable Root and Cleartext Logins

Once your keypair is working, (and you are able to log in with it) it’s time to eliminate root logins and cleartext logins. Some folks will tell you that root logins are fine with SSH because passwords don’t get sent in the clear. While that’s true, ‘root’ is still the one username that is guaranteed to be on every Unix-based machine, so if you are going to brute force an account, this is the one to focus your efforts on. Disabling root logins and clear text logins is all done in the sshd_config file:
sudo nano /etc/ssh/sshd_config

Press ‘ctrl+w’ and search for the word ‘root’. You are looking for this entry:
# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

Change ‘#PermitRootLogin yes’ to ‘PermitRootLogin no’. (uncomment if necessary and change from ‘yes’ to ‘no’.)

Then press ‘ctrl+w’ and search for the words ‘clear text’. You are looking for this entry:
# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

Change ‘#PasswordAuthentication yes’ to ‘PasswordAuthentication no’ (uncomment and change from ‘yes’ to ‘no’.)

Once these changes are made, DO NOT LOG OFF OF YOUR SSH SESSION. Once these changes are implemented, it will be hard to log back in to undo anything if you make a mistake. You should have tested and succeeded with your ssh-key based login because we are about to restart the ssh daemon and prevent clear text logins:
sudo systemctl restart ssh

UPDATE: on Debian 9 you can restart SSH with
service ssh restart
UPDATE: on CentOS 7 you can restart SSH with
systemctl restart sshd.service

To test ssh logins, connect to the IP of your server with putty using the ‘Default Settings’ profile. Your login attempt should fail because only people with private keys are allowed to the party:

At this point you are far from being hack-proof, but you are a bit more locked down than you were before, and there are always more convenient targets out there 🙂

Hardening web servers is another story, which really isn’t my bag to be honest. There’s a reason that I host my blogs with Google or WordPress 🙂

OH SHIT! I set this up on my Windows machine at home and I don’t have access to my private key at work/school/Aunt Tillie’s computer!

So you’ve set up your keys, you’ve disabled clear text logins and now you are trying to get a new public key onto your locked down box, but you can’t log in because you don’t have the private key. How screwed are you?

Not completely.

I find myself in this situation when I am setting up a new Linux or Unix workstation. With Windows I use a cloud storage service to keep my keys (that have passphrases), and I use one key pair per server, and just reuse my private key on each of my Windows machines.. The passphrase protects the private key on the cloud service (should the cloud service experience a breach or some other security failure) and on the local drive of my PC, should it fall into the wrong hands.

In the Unix world, I do the opposite. As I stated above, SSH keys are way easier to do with Unix. It’s no problem to produce key pairs and upload them to remote servers, so I use one key pair on per workstation, but then I use that pair on each of my servers.

This is probably a much safer practice than my Windows+Dropbox approach, but I use encryption tools like BitLocker and KeePass to add an little more security when I use Windows.

All you have to do on your new Unix box is create a new key pair like so:
ssh-keygen -t rsa -b 4096

And then use the ever so handy tool ssh-copy-id to add the newly created public key:
ssh-copy-id -i ~/.ssh/id_rsa user@remote-server

It couldn’t be easier, except when you have disabled password logons like an absolute walnut. In that case, you will need to log into both your new Unix box and your locked down remote Unix box from a computer that already has a key pair configured with the locked down remote server. For this reason, I recommend setting up a VM, a container, or a shell account to use as an intermediary.

By default, RSA key pairs will be stored in ~/.ssh/id_rsa (private) and ~/.ssh/id_rsa.pub (public). Once you are logged in on your third computer that is already set up with a key pair, open an SSH connection to the new Unix machine and an SSH connection to the locked down remote server. For the purposes of this demonstration, I will call the new box “newbox” and the locked down remote server “remotebox”. Both non-root user accounts will be called “steve”.

In the window for your session on newbox, view your id_rsa.pub file like so:
steve@newbox:~$ cat ~/.ssh/id_rsa.pub

You will see the familiar Very Long Line of Text ™ (VLLOT) which is the public key. The text will fill multiple lines. This is important to note, because you will need to copy all of these lines in a moment.

In the window for your session on remotebox, edit your authorized_keys file like so:
steve@remotebox:~$ nano ~/.ssh/authorized_keys

You will see the VLLOT that you pasted in when you locked down the server initially. It will be all on one line, and will most likely be truncated. This is important to note because you will be pasting text into this window in a moment.

Now you just copy the VLLOT from the window with your session on newbox into the editor window with your session on remotebox. If you are using putty for this operation, you can copy text by pressing ALT+C, instead of CTRL+C. CTRL+C cancels things in the Unix shell, and in Nano it will show the current cursor position.

Your window with our session on remotebox should now have two Very Long Lines Of Text(tm). Use the arrow keys on your keyboard to verify that the whole line is there. If you are certain that the VLLOT was pasted correctly, simply save the authorized_keys file and exit. In Nano, CTRL+O to saves and CTRL+X exits.

Once you have saved the authorized_keys file, switch to your window with your session on newbox. You can now attempt to connect to remotebox via SSH:

steve@newbox:~$ ssh steve@remotebox

If all goes well, you should be prompted for your passphrase for “/home/steve/.ssh/id_rsa”. If not, you probably didn’t paste the VLLOT correctly.

Public Access Unix Rocks

As I stated before, I recommend having a VM, container, or a Unix shell account to use as an intermediary for accessing locked down remote servers. Getting remote access to your gear is important. So important that I run multiple VMs to make sure that I can access everything remotely.

If you are not fortunate enough to be blessed with an embarrassment of hardware like I am, you can still use a dedicated server as an intermediary by signing up for a shell account with SDF.

I have had a shell account at SDF for decades. It enabled me to learn about large Unix systems without needing to set up a Linux box. Even if you have a server at home and a hosted server somewhere else, having a Unix shell account is still a great tool to have in your arsenal for way more than just stupid SSH tricks.

My .screenrc

I am a huge fan of screen. It’s indispensable for working on a Unix host via SSH. It lets me have multiple terminals (screens) up at a time. There are dudes that use screen to split their terminals into multiple views, like a tiling window manager, but for the command line.

My needs are not nearly as sophisticated, since I mostly use putty to connect to Linux servers from Windows.

I use 4 special keys:
F9 to detach from the screen session. This is leaves your session running in the background. I mostly use this to idle in IRC. Once detached from your session you can view your active screen session by typing:
screen -ls

Which will return something like this:

user@localhost:~$ screen -ls
There is a screen on:
2030.pts-0.localhost (05/25/2016 06:45:51 PM) (Detached)
1 Socket in /var/run/screen/S-user.


To reconnect to a detached screen session, type
screen -r 2030.pts-0.localhost

If the session is in use elsewhere, use the -D option:
screen -D 2030.pts-0.localhost

This will disconnect the screen session that’s in use, log off the SSH session that initiated it, and then reattach the active SSH session to the screen session.

Or, if you’re like me and never have any idea if you have a running a screen or not, just combine -D and -R and quit worrying about sockets and get on with your life:
screen -DR

And, if you are also like me and forget the switches for screen, just use the alias command in your .bashrc to have screen do -DR every time:
alias screen = 'screen -DR'

F10 to open a new terminal in screen.

This option lets you have multiple terminals in the same SSH session. This is handy for having a full screen app (like irssi) in one term, and one or more additional terms for running other commands. To close a terminal, type
exit

F11 and F12 to switch terminals
When you have multiple terminals open you can navigate them, from left to right with the F11 key to select the terminal to the right, and the F12 key to select the terminal to the left.

The File
To use this file, simply paste the contents below into a file called .screenrc in your home directory. So here it is, the .screenrc, that I have been using for years:

startup_message off

# Window list at the bottom.
# I got the long line of vars from https://bbs.archlinux.org/viewtopic.php?pid=423481#p423481
hardstatus alwayslastline
hardstatus string "%{.kW}%-w%{.W}%n %t%{-}%{=b kw}%?%+w%? %=%c %d/%m/%Y" #B&W & date&time

# From Stephen Shirley
# Don't block command output if the terminal stops responding
# (like if the ssh connection times out for example).
nonblock on

# Allow editors etc. to restore display on exit
# rather than leaving existing text in place
altscreen on

# bind F9 to detach screen session (to background)
bindkey -k k9 detach

# bind F10 to create a new screen
bindkey -k k; screen

# Bind F11 and F12 (NOT F1 and F2) to previous and next screen window
bindkey -k F1 prev
bindkey -k F2 next