Adventures in Proxmox Part 3: Chris don’t know shit about networking

When I first started messing with Proxmox, I crashed my home network.

I have since spent the last several months learning about Proxmox networking using virtual box. I have also been working on a parallel project: upgrading my home network to be segregated using VLANs. Like my budget for server hardware, my budget for network gear is practically nonexistent, so I have been doing a lot of reusing things that should have been replaced years ago.

After a bit of consternation, I settled on a prosumer router and a smart switch, rather than a PC-based router and a managed switch. Mostly because I needed this to work for the family as well as for the lab, and I didn’t want to spend weeks relearning Cisco. Time to tear down the old home network!!

A New Router

My plan is to have 4 “real” networks for my “physical” equipment:

  1. The family’s wireless network – for phones, tablets, game consoles, and tv sticks.
  2. My wired network for my personal workstations and servers.
  3. A VOIP network for POE phones, ATAs, and my PBX.
  4. A server and network lab for me to wreck things.

When I say “real” I really mean “operated by humans” or perhaps “not a Proxmox host”. When I say “physical” I also mean “operated by humans” or perhaps “not a Proxmox host”. At least half of these “real” ports are VLANs, and at least half of these “physical” devices are VMs. In this scenario, “real” and “physical” networks and devices are the ones that I and the family use, compared to the networks that are dedicated to the Proxmox cluster.

The critical distinction is that all of these network segments connect to a different port on the router, and have firewall rules to keep them from connecting to each other. In this scenario, a dumb switch plugged into each port of the router will provide a physically separated network at layer 2 (Ethernet) and a logically separated network at layer 3 (IP). It is here that I have used my first batch of dumb old mini switches:

  1. eth1 – Family Wireless, 192.168.10.0/24
  2. eth2 – Personal Wired, 192.168.11.0/24
  3. eth3 – VOIP, 192.168.12.0/24
  4. eth4 – Lab, 192.168.13.0/24

The family wireless network consists of 2 wireless access points, both with 4 dumb gigabit Ethernet ports:

  1. WAP port 1 -> eth1 on the router, uplink to the Internet
  2. WAP port 2 -> eth0 on the NAS appliance
  3. WAP port 3 -> port 1 on the smart switch
  4. WAP port 4 -> port 1 on the other WAP

So, I had my router set up, and plugging a laptop in to each dumb switch let me pull an IP from the DHCP server for the respective network segment. I was also able to browse the Internet. Awesome. I have managed to convert a big, clunky, error-prone network into four smaller error-prone networks. This is progress?

As far as the family is concerned, eth1 on the router is the network. Wireless access to both the Internet and to the data and media stored on the NAS. If I never plug in the smart switch then only I would notice. I have the WAP’s dumb switch plugged in to the smart switch because I have a media server VM on the Proxmox cluster that I want to put onto the wireless network to stream video to tablets, mobile phones and smart TVs. Because the cluster nodes only have 4 network ports, I need to put multiple network connections on to 1 of those network ports. This is where VLANs come into play. This is also where upgrading my knowledge of routing, switching, and firewalls comes in to play with Proxmox: putting the cluster onto all 4 of my network segments using just one network port from each node.

VLANs: everything you hate about dozens of dumb switches, plus virtualization

With the new router working, it’s time to configure the networks’ core: the smart switch.

VLANs are a great way to divide up a big physical switch into smaller virtual networks. A 24 port switch could be broken down into 4 networks, with a a varying number of ports in each network. You can also put a single switch port onto more than one VLAN. The network traffic gets put into the appropriate virtual network by using tags. You can even put a given port into “all” of the VLANs, this is sometimes referred to as a “trunk.” Trunks are used to connect multiple switches together, passing all tags between them.

Dumb switches can’t tag traffic. So, if you want to mix a smart switch that does VLANs with a dumb switch that doesn’t, you need to make sure that your untagged traffic is going out of the right ports. In the hypothetical 24 port managed switch in the example above, if you put port 2 into VLAN 2, and then plug a dumb switch into port 2, then port 2 needs to know what to do with untagged traffic. Traffic coming out of the dumb switch won’t have tags, and traffic going into to the smart switch will lose its tags. This is the essence of “VID” and “PID/PVID”. A VID is a VLAN ID, PVID is a Port VLAN ID. All the ports on the smart switch need to treat all traffic as tagged, even when it’s not. Untagged traffic needs to be treated differently than tagged traffic, basically meaning that “untagged” is just a special category of “tagged”. The PVID is a kind of “untagged == special tag” way for ports to deal with untagged traffic. This is the exact moment that I developed a migraine.

Star Trek guy with severe head pain.I have done a decent job keeping the family wireless network packets away from everything, and everything away from the family by putting each network segment on its own dumb switch. Now it is time to blur those boundaries a bit by plugging each of those dumb switches into the smart switch. My network is broken into 4 subnets, so my VLANs will break down something like this:

  • VLAN 1 – Family Wireless
  • VLAN 2 – Personal Wired
  • VLAN 3 – VOIP
  • VLAN 4 – Lab

I probably don’t need a separate /24 (class C) network for each VLAN, but I am not very clever and I have zero confidence in my ability to design networks or IP schemes. I know how routing works when you are using /24’s so for my implementation VLAN == /24. Also, as I learned in the Virtual Box lab, network designs get real confusing real fast, so having the VLAN tag roughly correspond to /24 subnet helps me to not go completely insane.

The smart switch is configured by a web interface. This interface has a default IP of 192.168.0.1, so I set a static IP on the Ethernet port of my laptop and logged in. This part of the configuration is important, and it will come into play again later. Once I have all the VLANs set up, I still need to be able to access the switch on this IP address.

I configured the first 4 ports on the switch as access ports or up-links to the dumb switches. Because the dumb switches don’t tag traffic, I needed the uplink ports to treat all “untagged” traffic as tagged to a single VLAN, using the PVID:

  • switch port 1 – VLAN 1, PVID 1
  • switch port 2 – VLAN 2, PVID 2
  • switch port 3 – VLAN 3, PVID 3
  • switch port 4 – VLAN 4, PVID 4

So now, if I change port 5 to VLAN 1 and PVID 1, I can plug in my Windows laptop and pull an IP from the wireless network. Then I can change port 5 to VLAN 2 and PVID 2, and now I can pull an IP from the wired network. Now I need to figure out how to get my Prox cluster nodes to sit on all 4 networks at the same time.

Enter the Management Workstation

Up to this point, I was able to set up my dumb switches and my VLANs with a Windows laptop. I just disabled the WiFi and plugged the Ethernet adapter into the various switches and ports. This was fine for scenarios where one switch port corresponded to just one network segment. But it turns out that Windows can’t do VLANs without proper hardware and software support for the NIC. If you have a VLAN-aware NIC and the Intel or HP enterprise app to configure it, I guess it works fine, but there is no Windows 10 app for the Intel NIC in my crashtop.

In my Virtual Box Proxmox lab, I learned that life is just easier when you have a Linux box dedicated to managing the cluster and testing your network setup, so I decided that before I set up the cluster, I should set up a “Management Workstation.” For the BoxProx lab, I used a Virtual Box VM running a GUI to administer the cluster because I needed a browser on the host only network. Technically, I could have run the management workstation without a GUI and just used SSH tunneling to access the web management interfaces for the Proxmox VMs, but I didn’t want to spend any time doing stupid SSH tricks. I also don’t have the actual hardware cluster running yet, so I need to do this with actual hardware. The hope is that once I get the VLANS and network bridges configured, the workstation will be superfluous. Therefore, the workstation doesn’t have to be powerful at all. Literally any old laptop or desktop that is laying around will do nicely.

My operating system of choice is Turnkey Linux Core. Set up an old desktop on port 5 of the smart switch. For the initial install, I left port 5 configured for VLAN 1 and PVID 1. I was able to pull an IP address from the wireless network, install and update the OS, and configure SSH.

Remote access is important because I can’t sit in my basement all day; Internet access is important because I need to install some network tools.

First step is to get the VLAN tools installed:

apt-get install vlan

Then enable VLAN support in the kernel:

echo 8021q | tee -a /etc/modules

Then add your tagged network interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

Then reboot the machine. I know there is a bunch of crap that you can do to avoid that, but this is the only way I can be sure that it works. I also know that if you name the interface eth0.N you probably don't have to mark the 'vlan-raw-device' but the Debian VLAN tutorial did it so I did it too.

What this does is change the IP of untagged interface eth0 to 192.168.0.10 (remember the IP of the switch from before?) and add eth0.1 (VLAN 1) with an IP of 192.168.1.10 and configured a default gateway and DNS for that interface.

Now, the machine should still be connected to the Internet, and you can modify port 5 on the smart switch to be in VLAN 1 and PVID 1.

If you can ping the IP for the smart switch (192.168.0.1), the IP of something on your wireless network (like an access point) as well as Google's DNS (8.8.8.8) then you are in good shape.

At this point, I left the basement and went upstairs. I used my laptop connected to the family wireless network (192.168.1.0/24) to SSH into the workstation. Since I will be making modifications to the smart switch configuration, as well as the management workstation, I decided to configure PuTTy to drop a local port and forward it to 192.168.0.1:80 so that I can access the web interface of the smart switch from my laptop, and the unencrypted HTTP traffic will be secured by the SSH tunnel.

Now I just need to move the Internet access to the 'Lab" VLAN and add the remaining VLANS to /etc/network/interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0

auto eth0.2
iface eth0.2 inet static
    vlan-raw-device eth0
    address 192.168.2.5
    netmask 255.255.255.0

auto eth0.3
iface eth0.3 inet static
    vlan-raw-device eth0
    address 192.168.3.5
    netmask 255.255.255.0

auto eth0.4
iface eth0.4 inet static
    vlan-raw-device eth0
    address 192.168.4.5
    netmask 255.255.255.0
    gateway 192.168.4.1
    dns-nameservers 8.8.8.8 8.8.4.4

The last step is to make sure that smart switch port 5 is part of VLANs 1, 2, 3, and 4. If all goes well, the workstation can ping the smart switch IP, Google DNS, and servers on all 4 VLANs.

The next step is to use this same network setup for the management NIC on the Proxmox cluster. Using the 4 VLAN interfaces for the network bridges (VMBR1-VMBR4).

Building a Proxmox Test Cluster in VirtualBox Part 5: Shit Happened; Lessons Were Learned

Jesus, it’s been almost a year since I posted part 1 of this series.

Hacking stuff is one of the ways that I cope with depression. Like going to the gym and getting stronger, learning new skills is a productive activity that improves my mind and my career. Also like going to the gym, hacking stuff requires a certain level of energy and focus. When I am having a depressive episode, I just can’t make myself do much more than watch TV. I have emerged from my Fallout 4 binge and I am eager to get this hardware cluster off the ground.

Learning Lessons

In my pursuit of a working Virtual Box + Proxmox cluster (Boxmox? ProxBox? BoxProx!) I discovered a few fatal flaws:

  • My testbed is a single laptop, and I used static IP’s that sat on my internal wireless network.
  • That meant that I could hack and demo the cluster at home, but not out in the world, like at Cinci2600.
  • Ergo, the “Management interface sitting on the internal network” question that I excluded from the exercise should not have been excluded.
  • Thus, the laptop-based lab for this project was missing a few things:
    1. 3 “Host Only” networks for the management interface, cluster network, and migration network.
    2. A router VBVM to route traffic bound for the Internet via a NAT interface.
    3. A management workstation VBVM with a GUI, for managing the router and the BoxProx CLI and UI.

The reason that I have been doing all of this in Virtual Box, is because it’s easy to recover from these sorts of mistakes. You can think of this exercise as the “Lab Before The Lab”, or the development phase, before going to an actual hardware lab. I actually gave up on keeping my lab environment separate from my home network because I was always limited by one thing or another. At this point, it’s as much lab as it is production, pretty much everywhere.

Shit Happening
Another component of this exercise that I have not documented is the redesign of my home/lab network to accommodate the new cluster. The old “cluster” is down to two old Proxmox servers that aren’t clustered together. It works for getting shit done for the family (PBX, Plex, Bittorrent, OpenVPN, etc.) but it’s not optimal, nor is the network sufficiently segregated to my satisfaction. So, as I have been doing this, I have also been upgrading the home network and learning more about things like VLANs.

So, the material of the first 4 parts of the series is valid, I just wanted to include the router and workstation bits, which you will probably only need if you want your lab to be portable, and work on wireless networks other than your home.

Modification to the network design

In the first installment, I recommended using a bridged adapter for the management interface. This worked great at home, but once I went anywhere else, the wheels fell off the whole process. I tried things like adding a static IP to my wireless adapter in Windows, and I came to the conclusion that Windows just doesn’t do virtual networking like it’s supposed to.

Hal turns on a light, but the bulb is broken. He takes a new light bulb from the shelf, but the shelf is also broken.

So, when you build your PVE hosts, use 3 host only networks, and use a router VM to connect the cluster to the Internet. Also be sure to disable the DHCP service on all of your host-only networks, like so:

The router

I know I have made simple routers from Debian VMs but for this experiment I spent a fair amount of time in the weeds. So do yourself a favor and just use PFSense. Yes it’s waaaay overkill for what you want to use it for, but it will route packets between two networks with minimal configuration, and that’s really what you want.

Hal gets a screwdriver to fix the shelf, and the drawer is squeaky. He picks up the WD40 but it's empty.

  1. Put the first interface of the PFSense VM on a NAT network.
  2. Make sure to disable the DHCP server on your host-only network interfaces.
  3. Put the second interface for the PFSense VM on the FIRST host-only network interface.
  4. Once you have the VBVM booted up, configure the WAN interface on the NIC that was configured by DHCP, and the LAN interface on the other NIC.
  5. Using the console on the router VBVM, configure the LAN for DHCP. Use a small address pool because there will probably be only one DHCP client ever. Using DHCP is an easy way to make sure that you are looking at the right NIC/virtual network.
  6. I can tell you from experience that if you find yourself twiddling with PFSense settings, you are doing it wrong. Just factory reset the config and move on. This is a BoxProx lab, not a PFSense lab.

The Workstation

Ok, so now you have a small network on host-only adapter 1, and router that connects it to the NAT network on your computer. All these NATs make the cluster network portable, but all but useless for anything else. That’s fine. All you want at this point is for your Linux workstation VBVM to access the Internet despite the fact that its only network interface is sitting on a host-only network.

Lois asks Hal to fix the light bulb and he is under the car yelling.

For the management workstation, you don’t need more than a browser and an SSH client, so literally any distro will work for you. I am a Debian guy, so when I want a no-frills GUI workstation with zero time spent configuring, I use one of the Ubuntu breeds meant for low end computers, like Lubuntu or Ubuntu Mate.

Regardless of the distro, you will be doing some repetitive typing in SSH. On Windows, I recommended MobaXTerm so you can type into multiple terminals at the same time and feel like a super hacker. In the Linux world, the app that you want to use is called “Terminator”. Like everything else on this blog, there is way more to Terminator that I won’t bother with. Just know that you can split your term into two equal parts horizontally and vertically by right clicking, and you can turn on and turn off broadcasting to all your keystrokes by pressing ALT+A and ALT+O respectively. Sorry Terminator/TMux/TWM fans, but I got shit to do.

This phase of the lab is a success if you can boot your Linux VBVM and use a browser to access Google as well as the web UIs for PFSense. You are now free to begin the lab again from Part 1.

Building a Proxmox Test Cluster in VirtualBox Part 3: Building The Cluster

In the last installment of this series, I discussed setting up the Proxmox VE hosts. Until now, you could do most of this configuring in triplicate with MobaXTerm. Now you can still use it to multicast, just be sure to disable it when you have to customize the configs for each host. This part of the process is a lethal combination of being really repetitive while also requiring a lot of attention to detail. This is also the point where it gets a bit like virtualization-inception: VirtualBox VMs which are PVE hosts to PVEVMs.

Network Adapter Configuration
I did my best to simplify the network design:

  • There are 3 PVE hosts with corresponding management IP’s:
    1. prox1 – 192.168.1.101
    2. prox2 – 192.168.1.102
    3. prox3 – 192.168.1.103
  • Each PVE host has 3 network adapters:
    1. Adapter 1: A Bridged Adapter that connects to the [physical] internal network.
    2. Adapter 2: Host only Adapter #2 that will serve as the [virtual] isolated cluster network.
    3. Adapter 3: Host only Adapter #3 that will serve as the [virtual] dedicated migration network.
  • Each network adapter plugs into a different [virtual] network segment with a different ip range:
    1. Adapter 1 (enp0s3) – 192.168.1.0/24
    2. Adapter 2 (enp0s8) – 192.168.2.0/24
    3. Adapter 3 (enp0s9) – 192.168.3.0/24
  • Each PVE hosts’ IP on each network roughly corresponds to its hostname:
    1. prox1 – 192.168.1.101, 192.168.2.1, 192.168.3.1
    2. prox2 – 192.168.1.102, 192.168.2.2, 192.168.3.2
    3. prox3 – 192.168.1.103, 192.168.2.3, 192.168.3.3

I have built this cluster a few times and my Ethernet adapter names (enp0s3, enp0s8, and enp0s9) have always been the same. That may be a product of all the cloning, so YMMV. Pay close attention here because this can get very confusing.

Open the network interface config file for each PVE host:

nano /etc/network/interfaces

You should see the entry for your first Ethernet adapter (the bridged adapter in VirtualBox), followed by the virtual machines' bridge interface with the static IP that you set when you installed Proxmox. This is a Proxmox virtual Ethernet adapter. The last two entries should be your two host only adapters, #2 and #3 in VirtualBox. These are the adapters that we need to modify. The file for prox1 probably looks like this:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

iface enp0s3 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.101
        netmask 255.255.255.0
        gateway 192.168.1.1
        bridge_ports enp0s3
        bridge_stp off
        bridge_fd 0

iface enp0s8 inet manual

iface enp0s9 inet manual

Ignore the entries for lo, enp0s3, and vmbr0. Delete the last two entries (enp0s8 and enp0s9) and replace them with this:

#cluster network
auto enp0s8
iface enp0s8 inet static
        address 192.168.2.1
        netmask 255.255.255.0
#migration network
auto enp0s9
iface enp0s9 inet static
        address 192.168.3.1
        netmask 255.255.255.0

Now repeat the process for prox2 and prox3, changing the last octet for the IP's to .2 and .3 respectively. When you are finished your nodes should be configured like so:

  1. prox1 - 192.168.1.101, 192.168.2.1, 192.168.3.1
  2. prox2 - 192.168.1.102, 192.168.2.2, 192.168.3.2
  3. prox3 - 192.168.1.103, 192.168.2.3, 192.168.3.3

Save your changes on each node. Then reboot each one:

shutdown -r now

Network Testing
When your PVE hosts are booted back up, SSH into them again. Have each host ping every other host IP to make sure everything is working:

ping -c 4 192.168.2.1;\
ping -c 4 192.168.2.2;\
ping -c 4 192.168.2.3;\
ping -c 4 192.168.3.1;\
ping -c 4 192.168.3.2;\
ping -c 4 192.168.3.3

The result should be 4 replies from each IP on each host with no packet loss. I am aware that each host is pinging itself twice. But you have to admit it look pretty bad ass.

Once you can hit all of your IP's successfully, now it's time to make sure that multicast is working properly. This isn't a big deal in VirtualBox because the virtual switches are configured to handle multicast correctly, but it's important to see the test run so you can do it on real hardware in the future.

First send a bunch of multicast traffic at once:

omping -c 10000 -i 0.001 -F -q 192.168.2.1 192.168.2.2 192.168.2.3

You should see a result that is 10000 sent with 0% loss. Like so:

192.168.2.1 :   unicast, xmt/rcv/%loss = 9406/9395/0%, min/avg/max/std-dev = 0.085/0.980/15.200/1.940
192.168.2.1 : multicast, xmt/rcv/%loss = 9406/9395/0%, min/avg/max/std-dev = 0.172/1.100/15.975/1.975
192.168.2.2 :   unicast, xmt/rcv/%loss = 10000/9991/0%, min/avg/max/std-dev = 0.091/1.669/40.480/3.777
192.168.2.2 : multicast, xmt/rcv/%loss = 10000/9991/0%, min/avg/max/std-dev = 0.173/1.802/40.590/3.794

Then send a sustained stream of multicast traffic for a few minutes:

omping -c 600 -i 1 -q 192.168.2.1 192.168.2.2 192.168.2.3

Let this test run for a few minutes. Then cancel it with CTRL+C.

The result should again be 0% loss, like so:

root@prox1:~# omping -c 600 -i 1 -q 192.168.2.1 192.168.2.2 192.168.2.3
192.168.2.2 : waiting for response msg
192.168.2.3 : waiting for response msg
192.168.2.3 : joined (S,G) = (*, 232.43.211.234), pinging
192.168.2.2 : joined (S,G) = (*, 232.43.211.234), pinging
^C
192.168.2.2 :   unicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.236/1.488/6.552/1.000
192.168.2.2 : multicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.338/2.022/7.157/1.198
192.168.2.3 :   unicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.168/1.292/7.576/0.905
192.168.2.3 : multicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.301/1.791/8.044/1.092

Now that your cluster network is up and running, you can finally build your cluster. Up to this point, you have been entering identical commands into a SSH sessions. At this point, you can stop using the multi-exec feature of your SSH client.

First, create the initial cluster node on Prox1, like so:

root@prox1:~# pvecm create TestCluster --ring0_addr 192.168.2.1 --bindnet0_addr 192.168.2.0

Then join Prox2 to the new cluster:

root@prox2:~# pvecm add 192.168.2.1 --ring0_addr 192.168.2.2

Followed by Prox3:

root@prox3:~# pvecm add 192.168.2.1 --ring0_addr 192.168.2.3

One final configuration on Prox1 is to set the third network interface as the dedicated migration network by updating the datacenter.cfg file, like so:

root@prox1:~# nano /etc/pve/datacenter.cfg

keyboard: en-us
migration: secure,network=192.168.3.0/24

Now that the cluster is set up, you can log out of your SSH sessions and switch to the web GUI. When you open the web GUI for Prox1 (https://192.168.1.101:8006) and you should see all 3 nodes in your TestCluster:

Now you can manage all of your PVE Hosts from one graphical interface. You can also do cool shit like migrating VMs from one host to another, but before we can do that we need to set up some PVEVMs. There are more things to set up, but to see them in action, we need to build a couple of PVEVMS to work with, which I will cover in the next installment: Building PVE Containers.

Building a Proxmox Test Cluster in VirtualBox Part 2: Configuring the Hosts

In the last installment of this series, I discussed setting up the Proxmox VE hosts in VirtualBox. At this stage in the exercise there should be 3 VirtualBox VMs (VBVMs) running, in headless mode. These VBVMs should be running Proxmox VE, ready to host their own VMs (PVEVMs).

Before you can set up the cluster, storage replication, and high availability, you need to do a bit of housekeeping on your hosts. In this post, I will go over those steps making sure that the hosts are up to date OS wise and that your storage is properly configured for later use. Most of these steps can be accomplished via the Web UI, but using SSH will be faster and more accurate. Especially when you use an SSH client like SuperPuTTY or MobaXTerm that lets you type and paste into multiple terminals at the same time.

Log in as root@ip-address for each PVE node. In the previous post, the IPs I chose were 192.168.1.101, 192.168.1.102, and 192.168.1.103.

I don’t want to bog this post down with a bunch of Stupid SSH Tricks, so just spend a few minutes getting acquainted with MobaXTerm and thank me later. The examples below will work in a single SSH session, but you will have to paste them into 3 different windows like a peasant, instead of feeling like some kind of superhacker:

Step 1 – Fix The Subscription Thing

No, not the nag screen that pops up when you log into the web UI, I mean the errors that you get when you try to update a PVE host with the enterprise repos enabled.

All you have to do is modify a secondary sources.list file. Open it with your editor, comment out the first line and add the second line:

nano  /etc/apt/sources.list.d/pve-enterprise.list

# old and busted
#deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

# new hotness
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

Save the file, and run your updates:

apt-get update; apt-get -y upgrade

While you are logged in to all 3 hosts, you might as well update the list of available Linux Container templates:

pveam update

Finally, if you set up your virtual disk files correctly according to the last post, you can set up your ZFS disk pool:

  1. List your available disks, hopefully you see two 64GB volumes that aren’t in use on /dev/sdb and /dev/sdc:
    
    root@prox1:~# lsblk
    NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                  8:0    0   32G  0 disk
    ├─sda1               8:1    0 1007K  0 part
    ├─sda2               8:2    0  512M  0 part
    └─sda3               8:3    0 31.5G  0 part
      ├─pve-swap       253:0    0  3.9G  0 lvm  [SWAP]
      ├─pve-root       253:1    0  7.8G  0 lvm  /
      ├─pve-data_tmeta 253:2    0    1G  0 lvm
      │ └─pve-data     253:4    0   14G  0 lvm
      └─pve-data_tdata 253:3    0   14G  0 lvm
        └─pve-data     253:4    0   14G  0 lvm
    sdb                  8:16   0   64G  0 disk
    sdc                  8:32   0   64G  0 disk
    sr0                 11:0    1  642M  0 rom
    root@prox1:~#
    

    Assuming you see those two disks, and they are in fact ‘sdb’ and ‘sdc’ then you can create your zpool. Which you can think of as a kind of software RAID array. There’s waaaay more to it than that, but that’s another post for another day when I know more about ZFS. For this exercise, I wanted to make a simulated RAID1 array, for “redundancy.” Set up the drives in a pool like so:

    
    zpool create -f -o ashift=12 z-store mirror sdb sdc
    zfs set compression=lz4 z-store
    zfs create z-store/vmdata
    

    In a later post we will use the zpool on each host for Storage Replication. The PVEVM files for each of your guest machines will copy themselves to the other hosts at regular intervals so when you migrate a guest from one node to another it won’t take long. This feature pairs very well with High Availability, where your cluster can determine if a node is down and spin up PVEVMs that are offline.

    Now that your disks are configured, it’s time to move on to Part 3: Building The Cluster.

Building a Proxmox Test Cluster in VirtualBox Part 1: Building The Hosts

In my last post, I set the stage for why I built the virtualbox cluster, and now it it time to discuss the how.

In researching the best way to design a network for a Proxmox cluster, the bare minimum is one network connection. This one link does the following:

  1. Hosts the web server for the management GUI – The web UI is pretty slick, and it’s great for viewing stats and checking for errors.
  2. Hosts the network bridge for guest VMs – This bridge acts as a kind of virtual network switch for your PVEVMs to talk to the outside world.
  3. Connects the host to the Internet – The PVE host needs to download security updates, Linux container templates, and install packages.

This one network interface is sort of the lifeline for a Proxmox host. It would be a shame if that link got bombed by incessant network traffic. As I discovered (the hard way) one possible source of incessant network traffic is the cluster communication heartbeat. Obviously, that traffic needs to go on its own network segment. Normally, that would be a VLAN or something, but I have some little dumb switches and the nodes have some old quad port NICs, so I wanted to just assign an IP to one port, and plug that port into a switch that is physically isolated from “my” network.

Once a cluster is working, migrating machines happens over the cluster network link. This is OK, but if your cluster network happens to suck (like when some jackass plugs it into a 10 year old switch) it can cause problems with determining if all the cluster nodes are online. So, now I want to set up an additional interface for VM migration. Migration seems like the kind of thing that happens only occasionally, but when you enable Storage Replication, the nodes are copying data every 15 minutes. Constant cluster chatter, plus constant file synchronization, has the potential to saturate a single network link. This gets even worse when you add High Availability, and there is a constant vote on if a PVEVM is up and running, followed by a scramble to get it going on another node.

So, at minimum we will need 3 network interfaces for the test cluster on VirtualBox. I didn’t want to spend a lot of time tinkering with firewall and NAS appliances, so I am leaving the “Prox management on its own network segment” and the “Dedicated network storage segment” discussions out of this exercise. I can’t decide if the management interface for my physical Proxmox cluster should sit on my internal network, or on its own segment. For this exercise, the management interface is going to sit on the internal network. My Synology NAS has 4 network ports, so I am definitely going to dedicate a network segment for the cluster to talk to the NAS, but that won’t be a part of this exercise.

[Virtual] Hardware Mode(tm)

Once you are booted up and VirtualBox is running, you can start building your VBVMs. I recommend building one VBVM to use as a template and then cloning it 3 times. I found that I kept missing important things and having to start over, so better to fix the master and then destroy the clones.

I called my master image “proxZZ” so it showed up last in the list of VBVMs. I also never actually started up the master image, so it was always powered off and the ZZ’s made it look like it was sleeping.

Create proxZZ with the following:

  • First, make sure that you have created 2 additional Host Only Network Adapters in VirtualBox. In this exercise you will only use two, but it can get confusing when you are trying to match en0s9 to something, so do yourself a favor and make three. Make sure to disable the DHCP server on both adapters.
  • Create a new virtual machine with the following characteristics :
    1. Name: ProxZZ
    2. Type: Linux
    3. Version: Debian 64bit (Proxmox is Debian under the hood.)
    4. Memory Size: 2048MB
    5. Hard drive: dynamically allocated, 32GB in size.
  • Make sure that you have created 3 total virtual hard disks as follows:
    1. SATA0: 32GB. This will be your boot drive and system disk. This is where Proxmox PVE will be installed. Static disks are supposed to be faster, but this isn’t even remotely about speed. My laptop has a 240gb SSD, so I don’t have a ton of space to waste.
    2. SATA1: 64GB, dynamically allocated. This will be one of your ZFS volumes.
    3. SATA2: 64GB, dynamically allocated. This will be your other ZFS volume. Together they will make a RAID1 array.
  • WHile you are in the storage tab, make sure to mount the Proxmox installer ISO
  • Make sure that you have created 3 network interfaces as follows:
    1. Adapter 1: Bridged Network – this will be your management interface.
    2. Adapter 2: Host Only Network Adapter #2 – this will be your cluster interface.
    3. Adapter 3: Host Only Network Adapter #3 – this will be your VM migration interface.
    4. You may be tempted to do something clever like unplugging virtual cables or something. Don’t. You will be cloning this machine in a minute and you will have a hard time keeping all of this straight.
  • Before you finish, make sure that the machine is set to boot from the hard drive first, followed by the CD/Optical drive. This seems stupid, but you will be booting these things in headless mode, and forgetting to eject the virtual CD rom is super annoying. So fix it here and stop being bothered with it.

When it’s done, it should look something like this:

Once you are sure your source VM is in good shape, make 3 clones of it. Don’t install Proxmox yet. SSH keys and stuff will play a major role in this exercise later, and I am not sure if VirtualBox is smart enough to re-create them when you clone it. I ran into this a few times so just clone the powered off VBVM. I called the clones prox1, prox2, and prox3.

[Virtual] Software Mode(tm)

Now it is time to start your 3 clones. This can get pretty repetitive, especially if you start the process over a couple of times. While you will appreciate cloning the servers, there isn’t really a simple way that I have discovered to mass produce the PVE hosts. In a few iterations of this exercise, I misnamed one of the nodes (like pro1 or prx2) and it’s super annoying later when you get the cluster set up and see one of the nodes named wrong. There is a procedure to fix the node name after you build it, but seriously just take your time and pay attention.

When you first start the install you will probably see an error about KVM Virtualization not being detected. Which should be impossible, right? The PVE host is *literally* a virtual machine. This is probably VBox for Windows not passing something to PVE Linux, or VBox misidentifying your CPU. Whatever it is, this isn’t a showstopper because our test systems that will run on the PVE cluster aren’t going to be VMs, they’ll be Linux Containers, but in the interest of not making my head hurt I am still going to call them VMs. The PVE hosts are VirtualBox VMs, and the Linux Containers running on Proxmox are Proxmox VE VMs. VBVMs are VMs running under VirtualBox, PVEVMs are Linux Containers running under Proxmox VE. *whew*

As you do the install, select your 32gb boot drive and configure your IP addresses.
I went with a sequence based on the hostname:
prox1 – 192.168.1.101
prox2 – 192.168.1.102
prox3 – 192.168.1.103
Like I said before, go slowly and pay attention. This part is super repetitive and it’s easy to make a stupid mistake that you have to troubleshoot later. At some point, I guarantee that you will give up, destroy the clones, and start over 🙂

Send In The Clones

Once your hosts are installed, it’s time to shut them down and boot them again, this time in headless mode. This is where fixing the boot order on ProxZZ pays off. With all 3 VBVMs are started up, you are ready for the next stage of the exercise: installing your PVE hosts.

Upgrading a hosted Debian 8 VM to Debian 9

A long time ago, I extolled the virtues of Cloud at Cost’s developer cloud. It’s a good tool for spinning up a box to mess with, but it’s far from being reliable enough for “production” use. What it is great for is having a box that isn’t constrained by a network (like a VM at work might be), but for which access to it may require modifications to a local firewall (like a VM at home might be), while avoiding the cost of a “real” production VM on Digital Ocean or Amazon.

Using a VM this way is a bit like building your house out of straw. It goes up fast, but it comes down fast too. So I have gotten used to setting up machines quickly and then watching them be corrupted and blowing them away.

Sometimes I do something stupid to corrupt them, sometimes they go corrupt all on their own.

The base Debian install on C@C is getting a bit long in the tooth, so part of my normal setup is now upgrading Debian 8.something all the way to Debian 9.whatever. This procedure will take a pretty long time. A long enough time that you will probably have to leave home to go to work, or vice versa. I recommend locking down SSH and then installing screen so you can attach and detach sessions that are running in the background.

Step 1 – update your sources and update your current version:

First, you should check your version of Debian, to make sure that you are on some version of Jesse, and not an earlier version for some reason:

# cat /etc/debian_version

The sources on a brand new C@C Debian 8 box are woefully out of date. Use your favorite editor (mine is nano; fight me) to edit the sources list.


# nano /etc/apt/sources.list

### Remove the entries and paste these in ####
deb http://deb.debian.org/debian/ jessie main contrib non-free
deb-src http://deb.debian.org/debian/ jessie main contrib non-free

deb http://security.debian.org/ jessie/updates main contrib non-free
deb-src http://security.debian.org/ jessie/updates main contrib non-free

Once you have the list updated, save the file and run the upgrade scripts like so:

# apt-get update
# apt-get upgrade
# apt-get dist-upgrade

On a new install this will take a long time. Note that if you are having trouble installing screen or fail2ban, you probably have to do this step before installing them.

Step 2 – See how bad the damage is

Now we see what kind of hell we will be unleashing on this poor little machine by upgrading just about everything. First, see what packages are broken:

# dpkg -C

On a fresh debian 8 box, there shouldn’t be a lot to report. If there is you need to fix those packages. Assuming that you got no messages about messed up packages, you can see what’s been held back from upgrade like so:

# apt-mark showhold

If you got a message that packages are obsolete, you can remove them like so:

# apt-get autoremove

Hopefully you don’t have any messed up packages, and you can proceed to the next step.

Step 3 – Do the thing

Now it’s time to change the sources from Jesse to Stretch and basically do step 1 all over again.

First you update the sources.list file:


# nano /etc/apt/sources.list

### Remove the entries and paste these in ####
deb http://httpredir.debian.org/debian stretch main
deb http://httpredir.debian.org/debian stretch-updates main
deb http://security.debian.org stretch/updates main

And then tag packages to be updated:

# apt-get update

Now do a dry run to see if it will blow up or not:

# apt list --upgradable

Assuming there are no flashing red lights or whatever it’s time to pull the trigger.

Step 4 – Hold on to your butt

Once you run the next set of commands, you will be asked if you want to restart services without asking. Assuming that you are doing this in screen, you can lose your SSH connection and the process will still run. In the event of a catastrophic failure, you can probably open the console and attach to your screen session, so say yes and then buckle up.

TIMES UP! LET’S DO THIS! LEEEEEEEEEERRRRROOOOOOOYYYYY:

# apt-get upgrade
# apt-get dist-upgrade

This will take a long time. Like a really long time. It’ll look cool tho. Having a command line window with text rolling by always makes me feel like Neo from the Matrix.

Step 5 – ??? Profit

Once it’s done, check the Debian version and revel in your victory:

# cat /etc/debian_version

Then check for obsolete packages, for which there will probably be a bunch:

# aptitude search '~o'

And then finally remove them all, like so:

# apt-get autoremove

Just to be safe, you should probably update and upgrade one last time:

# apt-get update
# apt-get upgrade

Step 6 – Diversify your backups

Now that you have gone through all of the difficulty of upgrading your house made of straw, it would be a shame for a big bad wolf to blow it down. For this reason, I recommend an old school Unix backup with tar, and keeping a copy of your backup on another computer. For this second part we will be using scp, and I recommend setting up SSH Keys on another Unix host. This might be a good time to set up ssh key pairs without passphrases for your root accounts to use.

The security model looks something like this:

  1. No one can log into any of the hosts via SSH as root.
  2. No one can log into any of the hosts without a private key.
  3. Your plain user account’s private key should require a passphrase.
  4. Your root password should be super strong, probably randomly generated by and stored in password manager like KeePass.
  5. If you want to scp a file as root without a passphrase, you should have logged in as a plain user with a private key with a passphrase and then used su to become root.
  6. If you can get past all those hurdles, a second public key passphrase isn’t going to protect much.

Change to the root of the file system (/) and run a giant compressed backup job of the whole filesystem (except for the giant tarball that you are dumping everything into).

# cd /
# tar -cvpzf backup.tar.gz --exclude=/backup.tar.gz --one-file-system / 

This will also take a long time, so you should seriously be using screen. Also, there is a lot of stuff in the backup that doesn’t actually need to be backed up, so you could add additional –exclude=/shit/you/dont/need statements to shrink the size of your backup file.

Once once the backup is done you can then change the name of the backup file to that of your machine name and use SCP to copy off the backup file to another Unix host. In the example below I am calling the backup randoVM. You should change the name because you may be backing up multiple VMs to the same source. I like to use my HUB VM at home because it has a lot of [virtual] disk space compared to my hosted VMs.

# mv /backup.tar.gz /ranoVM.tar.gz
# scp randoVM.tar.gz steve@home.stevesblog.com:~/backups/ranoVM.tar.gz 

You can leave the big tarball on your VM’s file system, or you can delete it. There are merits to doing either. You will want to repeat this backup procedure periodically as you add features and services to the VM.

If you find yourself needing to restore the VM because you or the big bad wolf did something stupid, you can simply pull the backup down and expand it.

# cd /
# scp steve@home.stevesblog.com:~/backups/ranoVM.tar.gz .
# tar -xvpzf ranoVM.tar.gz

Turnkey Torrents and Swedish Internets

A few months ago, I wrote about using a Turnkey Linux OpenVPN appliance to route network traffic thru Sweden. Since that time I have gotten my BitTorrent machine running. The other post was mostly about getting the VPN tunneling and routing to work. This post will mostly be about setting up the torrent server.

The Turnkey Torrent Server is neat because it’s a minimal Debian machine with a pre-configured Transmission BitTorrent Client, a web interface for managing BitTorrent, a working Samba server, and a webDAV client so you can use a browser to download files. Basically, you use the web interface to grab things, the Samba server to makes them accessible to your media players on your internal network, and webDAV makes the files accessible to the rest of the world, assuming you have the right ports forwarded. My preferred method for watching torrented videos is on a PC plugged into a TV running VLC Media player controlled with a wireless keyboard. I know I should be using Plex and shit like that, but I’m old school.

The Swedish Connection
For some of my friends who aren’t pirates (especially the friends that are into British TV) I am like their coke dealer except I deal in movies and TV shows. That means that sometimes I get asked to find things when I’m not at home. Like a third of my remote access shenanigans, A.K.A. reverse telecommuting, is so that I can pull up BitTorrent and snarf shit for friends and relatives when I’m not at home. Being able to expose the uTorrent remote interface to the web was great for letting my more technical non-hacker friends grab torrents without any assistance from me.

My VPN provider gives me the option of forwarding ports. When I was running uTorrent on a dedicated Windows machine, those forwarded ports were easy to configure. I would just set them up on the VPN site and map them to the ports I configured in uTorrent. One was for BitTorrent transfers to make sure that my ratios reported correctly on private trackers. The other was for the uTorrent web interface. For a long time I ran Windows for torrenting because I used PeerBlock to help me fly under the radar. Times change tho. Real time block lists is old and busted. VPNs is the new hotness. Unfortunately, with this VPN router setup it messes up forwarding ports. When I set up port forwarding on the VPN provider side, the forwarded ports hit the doorway server rather than the torrent server, so that has to be fixed with more IPTables kung fu on the doorway server.

I know I said that I wasn’t going to write anymore about the doorway server, but I lied. I needed to configure the doorway server to open those ports and then forward them to the torrent server. Let’s assume that my internal network is a 192.168.1.0/24 subnet (a class A block, a range of addresses from 192.168.1.1 to 192.168.0.254) with a default gateway of 192.168.1.1. All of my traffic goes through my local router and hits the Internet from my ISP, in the US. If a device asks for an IP via DHCP, this is the network configuration that it will receive, along with red-blooded American Internets. Here is an awful network diagram because why not?

The doorway server has a static IP of 192.168.1.254 and it’s configured to route all of its traffic through the VPN tunnel to Sweden. Any device that is configured to have a default gateway of 192.168.1.254 will also hit the Internet via the tunnel to Sweden, thereby receiving Swedish Internets. At this point, all the configuration is done, and your torrent server will work, but there won’t be any ports forwarded to it, which is lame. No forwarded ports is especially lame when you are using private trackers because it can really mess with your ratios. Now, you could just open a port on your firewall for the web interface on the American side, but that’s also pretty lame. If you want to use your torrent server, you should strictly be using Swedish Internets.

Welcome to Swedish Internet
To forward those ports, first set them up in Transmission, then with your VPN provider. The web interface port [12322] is already configured for you by Turnkey Linux. You can set the other port in the Preferences->Network->Listening Port blank. Once the entry points and the end points are configured, it’s time to do more iptables kung fu.

Let’s assume the following:

  1. The web interface port for Transmission is 12322.
  2. The listening port in Transmission to 9001.
  3. The static IP for your torrent server is 192.168.1.10
  4. The doorway server IP is 192.168.1.254.
  5. The forwarding ports you were able to get from your VPN provider are 9000 and 9001.
  6. You want to use port 9000 on the VPN side for the Transmission web interface.
  7. You wand to use port 9001 on the VPN side for the Transmission listening port.

What needs to happen is for the VPN tunnel interface (tun0) to listen on ports 9000 and 9001, then forward traffic on those ports to 192.168.1.10. Then, you want any traffic on those same ports that comes from the doorway’s internal network interface (eth0) to be modified so that it doesn’t look like it came from the tunnel interface. This is super important for TCP handshakes.

First create your rules for accepting/forwarding connections on the VPN side:


iptables -A FORWARD -i tun0 -o eth0 -p tcp --syn --dport 9000 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -i tun0 -o eth0 -p udp --dport 9001 -m conntrack --ctstate NEW -j ACCEPT

This was probably configured fine in the doorway server post, but this specifically allows all the traffic that passes between your VPN and the local network connections once a connection is establshed:


iptables -A FORWARD -i eth0 -o tun0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i tun0 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Now add the rules to rewrite packets destined to the web interface and then rewrite the responses:


iptables -t nat -A PREROUTING -i tun0 -p tcp --dport 9000 -j DNAT --to-destination 192.168.1.10:12322
iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 9000 -d 192.168.1.10 -j SNAT --to-source 192.168.1.254

Add the rules to rewrite all the BitTorrent packets, including responses:


iptables -t nat -A PREROUTING -i tun0 -p udp --dport 9001 -j DNAT --to-destination 192.168.1.10:9001
iptables -t nat -A POSTROUTING -o eth0 -p udp --dport 9001 -d 192.168.38.37 -j SNAT --to-source 192.168.1.254

All the strict rewriting probably isn’t a big deal for the BitTorrent traffic because it’s UDP, and UDP don’t give a fuck.

If it’s working, point your browser to https://the-ip-address-of-your-vpn-server:9000 and you should be prompted to log in to the web interface. Once you’re sure it’s all good, then it’s time to save your working iptables config:

iptables-save | tee /etc/iptables.up.rules

Make sure that your rules work well after you reboot your VM. And then run your backups to make sure that they have your latest config because there’s nothing worse than trying to piece all this crap together for the third time.

You can skip having to remember the IP by registering it as a subdomain somewhere, either with a dynamic DNS service, or with the registrar for a domain that you own.

In the unlikely event that I made this, or any other technical thing look easy, rest assured that it took me at least a couple hours. Also, I had it working a months ago, but I forgot to update my snapshot and had to redo it again because I am not a smart man. Then during this second go around I had to restore the VM from a backup because iptables just isn’t my bag. Thankfully BitTorrent is my bag. Happy pirating!

Adventures in Proxmox Part 1: Words About Boxes

The Proxmox logo
It’s been a few weeks since I exorcised HyperV from my life like an evil demon. I have replaced it with Proxmox and so far it’s been mostly great. With a couple of serious caveats.

Plastic dinosaurs betraying each other.My transition to Proxmox has been a rather involved, not so much because Proxmox is hard to set up (it’s not), but because I am tired of slapping old junky hardware together and hoping it doesn’t die, and then scrambling to fix it when it inevitably betrays me. Unlike most dudes with home servers and labs, most of my acquisitions were made years ago to support an MMO habit. Specifically multiboxing.

PC case made from peg board.

I call them “computers” because they are computers in the sense that they have CPU’s, RAM, and HDD’s. But they were low-budget things when they were assembled years ago. The upgrade path works something like this:

  1. A computer begins its life as my main gaming machine that will run my favorite game at a satisfactory speed and resolution.
  2. Then I find a new favorite and upgrade the gaming machine’s guts to run the new game.
  3. The old gaming guts get transplanted in to my “server” where they are *barely* able to run a few VMs and things like that.
  4. The final stage is when the server guts are no longer up to the task of running VMs. I then add a few old network cards and the “server” then becomes my “router”.
  5. The old router guts then get donated somewhere. They’re not really useful to anyone, so they probably get shipped to Africa where they get mined for gold and copper by children at gunpoint.

Breaking the [Re]Cycle of Violence
Wall-E holding a pile of scrapIn the years since then, I have taken to playing epic single player games like Skyrim. These games really only need one machine. The rest of the gear I used to run little “servers” for one thing or another, which I have slowly replaced with VMs. The problem with using old junky computers as servers is when you run them balls out 24 hours a day. In my search for a replacement VM host, I spent a lot of time researching off-lease servers. My goal was to have 8 cores and 32gb of ram, with the ability to live migrate VMs to another [lesser] host in an emergency, something that my HyperV setup was lacking. After a lot of consternation, I decided that since a single VM would never actually use more than 4 cores or 8gb of RAM, why not use 2 [or more] desktops?

A room full of old PCs.I found some old off-lease quad-core Intel desktops for about the same retail price as a low end server processor. I used the RAM from my older gaming machines/VMservers and some hard drives from some old file servers to build out my “new” Proxmox cluster. With two quad core desktops running maxed-out memory(16GB each) I managed to satisfy my need to be like the other kids with “8 cores with 32GB of RAM” for about the price of an off-lease server chassis, with the added bonus having a cluster. The goal is to add nodes to grow the cluster to 16 cores and 64GB of RAM, while also adding clustered storage via Ceph to make use of old hard drives from file servers.

New hot servers is old and busted. Old busted clusters is the new hotness.
For me, the clustered model is better, in my opinion for a number of reasons. It mostly has to do with modularity:

  1. You can build out your infrastructure one paycheck at a time. Part of the problem with off-lease servers is that while the chassis is cheap, the components that go in it are expensive and/or hard to find. The deal with servers is that the cost of the motherboard and CPU are nothing compared to what you will spend on RAM. I was looking for something I could start using for less than $200, and a refurb desktop and RAM from old gaming boxes got me going at that price point.
  2. Desktops stack on top of each other for free. I don’t have any server or telco racks, so in addition buying ECC RAM, I would also be buying a rack, rails, and all of the other stuff that goes with them. This would easily eat up my $200 startup budget before I powered on a single box.
  3. Moar boxes == moar resiliency. My gear at home is part lab and part production environment. Yes, I use it to hack stuff and learn new things, but my family also uses it in their daily lives. Network shares stream cartoons; VOIP phones connect friends; keeping these things going is probably as important as my day job. Being able to try bold and stupid things without endangering the “Family Infrastructure” is important to my quality of life.
  4. Scaling out is probably more important than Scaling Up. A typical I.T. Department/Data Center response to capacity problems is to regularly stand up newer/more powerful [expensive] gear and then dump the old stuff. I guess this is a good approach if you have the budget. It certainly has created a market for used gear. I don’t have any budget to speak of, so I want to be able to increase capacity by adding servers while keeping the existing ones in play. There are still cost concerns with this approach, mainly with network equipment. In addition to upping my server game, I am going to have to up my networking game as well.

It works…ish

I have my two cluster nodes *kind of* working, with most of my Linux guests running as containers, which is very memory and CPU efficient. I am running two Windows VMs, PORTAL for remote access and dynamic DNS, and MOONBASE which I am using for tasks that need wired network access. All of my desktops are currently in pieces, having donated their guts to the “Cluster Collective” so I am mostly using my laptop for everything. I am not really in the habit of plugging it in to Ethernet, or leaving it turned on, so for now I am using a VM in place of my desktop for long running tasks like file transfers.

I say that the cluster is only kind of working because my home network isn’t very well segmented and the cluster heartbeat traffic straight up murders my little switch. It took me a while to figure out the problem. So the cluster works for a few days and then my core switch chokes and passes out, knocking pretty much everything offline. For now, the “cluster” is disabled and the second node is powered off until my new network cards arrive and I can configure separate networks for the clustering, storage, and the VMs.

Coming soon: Adventures in Proxmox part 2: You don’t know shit about networking.

Additional Remote Access Shenannegans

In my previous post, I expanded on my preferred methods for gaining remote access to my home network. Since then, I have decided to quit using Hyper-V because it’s awful.

I have now decided to move to ProxMox on my server. Proxmox is pretty cool, although the documentation sucks. I recently started using Linux containers for my remote access servers instead of VMs, which ProxMox supports out of the box. A truly compelling feature of Proxmox is its integration with Turnkey Linux. You can download Turnkey Linux Container Templates directly in Proxmox and spin them up quickly. I used the Turnkey OpenVPN template to rebuild GATE, my OpenVPN server.

The performance improvement is remarkable. On Hyper-V, each Linux VM ate 512MB of RAM just to sit idle 99.9% of the time. So far I have 3 containers configured with 512MB of ram each, but they use roughly 25-50MB each and they leave the rest for the server. PORTAL, my Windows VM, still takes his share of the RAM and doesn’t give it back, but that’s nothing new.

Moar RAM == moar servers!
On the plus side, efficient use of memory means that I can feel better about running a dedicated Linux box (container) for each application. Dedicated boxes mean that when I inevitably screw something up, it doesn’t affect the other applications that are running (that I haven’t screwed up yet.) Also, with pre-built containers and snapshots, you can toss machines that you screwed up without losing much time. I know, I know, rebuilding a Linux box instead of fixing it is sacrilege… but I got other shit to do.

On the minus side, containers don’t really act like VMs, especially when it comes to alternative network configurations. In particular, a Linux Container that uses a TUN or TAP interface needs some extra configuring. The TUN interface is how OpenVPN does its thing, so getting my GATE machine, the OpenVPN server that allows access to the DMZ on my internal network took a lot of fiddling with to get right. I did a bunch of Googling and I ended up with this forum post that recommends rebuilding the TUN interface at boot time with a script.

Here is the TUN script that I have graciously stolen so that I don’t have to Google it again (I didn’t even bother to change the German comments):

#! /bin/sh
### BEGIN INIT INFO
# Provides:          tun
# Required-Start:    $network
# Required-Stop:     $openvpn
# Default-Start:     S 1 2
# Default-Stop:      0 6
# Short-Description: Make a tun device.
# Description:       Create a tundev for openvpn
### END INIT INFO

# Aktionen
case "$1" in
    start)
        mkdir /dev/net
        mknod /dev/net/tun c 10 200
        chmod 666 /dev/net/tun
        ;;
    stop)
        rm /dev/net/tun
        rmdir /dev/net
        ;;
    restart)
        #do nothing!
        ;;
esac

exit 0

Then you enable the script and turn it on:
chmod 755 /etc/init.d/tun
update-rc.d tun defaults

With this script, I was able to stand up a real OpenVPN server (not just an Access Server appliance) for unlimited concurrent connections! Not that I need them. I’m the only one that uses the VPN and most of the time I just use SSH tunnels anyway.

Since OpenVPN container templates make standing up servers so easy, I thought I’d build another one that works in reverse. In addition to GATE that lets OpenVPN clients route in to the DMZ, I thought I would use an OpenVPN client to route traffic from some DMZ hosts out to the Internet via Sweden. In the past, I used a VPN service to dump my Bittorrent box’s traffic this way, but I would like to extend that service to multiple machines. EVERYBODY GETS A VPN!

Öppna dörr. Getönda flörr.
I couldn’t figure out what a machine that does this kind of thing is called. It’s a server, but it serves up its client connection to other clients. It’s a router, but it just has the one network interface (eth0) that connects to a tunnel (tun0). It’s basically setting up a site-to-site VPN, but the other site is actually a secure gateway. This identity crisis led to a terminology problem that made finding documentation pretty tough. Fortunately, I found another pirate looking to do the same thing and stole his scripts 🙂

Since it’s a doorway to a VPN gateway to Sweden, I decided to call the box DÖRR, which is Swedish for “door”. I did this to maintain my trans-dimensional gateway theme (HUB, GATE, PORTAL, etc.)

Also, I would like to apologize to the entire region of Scandinavia for what I did you your languages to make the pun above.

The Turnkey Linux OpenVPN template sets up in one of 3 modes: “Server”, “Gateway”, or “Client”. “Server” is the option I went with for GATE, which allows OVPN clients the option of accessing local subnets. This is the “Server” portion of a Site-to-Site VPN or a corporate VPN. “Gateway” forces all OVPN clients to route all traffic through it, this is the config for secure VPN services like NordVPN or AirVPN. “Client” makes a client connection to another OVPN server. If you connect a “Client” to a “Server” you get the full Site-to-Site solution, but there is no documentation on Turnkey about setting up a “Site-to-Site Client” to route traffic from its internal subnet to the “Site-to-Site Server”.

What I am looking to do is configure a “Site-to-Site Client” but point it to a “Gateway”. Another important consideration when setting this up was that I didn’t want to do any meddling with the setup of my DMZ network. I just want to manually configure a host to use DÖRR as its default gateway. No need for proxies, DNSMasq, DHCP or anything like that. Just static IP’s, the way God intended it 🙂

Step 1 – The Site-to-Site Client
Once I got the container running, I had to fix the /dev/tun problem (the script above) and then make some config changes to OpenVPN.

Because this is a VPN client, and not a server, you need to get the OpenVPN client profile loaded. The bulk of my experience with OpenVPN clients is on Windows where you start the client when you need it. For this application you need to automatically run the OpenVPN connect process at boot and keep it running indefinitely.

First, you need to obtain a client config. I downloaded my ‘client.ovpn’ file from my VPN provider, and I copied it to /etc/openvpn/client.conf as root. You can name the files whatever you want, just remember what you named them because it’s important later.

cp /root/client.ovpn /etc/openvpn/client.conf

Now test the connection to make sure everything worked

openvpn --config /etc/openvpn/client.conf &

The & is important because it puts the OpenVPN process into the background, so that you get your command prompt back by pressing ENTER a couple of times. You can then test your Internet connection to see what your IP is a few different ways. You can use SSH with a dynamic port and tunnel your web traffic thru it with a SOCKs proxy. You could use curl or lynx to view a page that will display your IP. Or you could just use wget. I like to use ifconfig.co like so:

wget -qO- ifconfig.co

If all goes well, you should see your VPN provider’s IP and not your ISP’s.

Once you get the VPN client working, you then want it to start up and connect at boot time. You do this by setting the ‘autostart’ option in /etc/default/openvpn.

nano /etc/default/openvpn
AUTOSTART="client"

If you changed your ‘/etc/openvpn/client.conf’ filename, you change the name here. The AUTOSTART value is the name of that file minus the ‘.conf’

Now reboot your server and do your wget test again to make sure that the VPN connection is starting automatically.

Once that is working, you have to route traffic. This means IPTables, because OpenVPN and IPTables go together like pizza and beer.

Step 2 – De Routningen

Normally to route traffic between interfaces on Linux, you have to add IP forwarding (echo 1 > /proc/sys/net/ipv4/ip_forward etc.) In this case, the Turnkey OpenVPN template has already done that for you. All you have to do add a few forwarding rules:

iptables -A FORWARD -o tun0 -i eth0 -s 192.168.1.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A POSTROUTING -t nat -j MASQUERADE

Now it’s time to test them. For this you need a client computer with a static IP. For the default gateway you want to use the static IP that you assigned to eth0 on your VPN doorway server. I used 192.168.0.254 for DÖRR. If your test box also shows your VPN provider’s IP when you access a site like ipleak.net then it’s time to make those rules permanent. By saving them to /etc/iptables.up.rules. It is important to save them to that specific file because the Turnkey template calls that file when setting up the eth0 interface in /etc/network/interfaces.

iptables-save | tee /etc/iptables.up.rules

I don’t know why it’s set up that way. I’m just here to make awful jokes about Germanic languages.

Once that’s done, reboot the doorway server one last time and test with your client computer with the alternate default gateway.

Now that the my VPN client is working again, I need to rebuild my BitTorrent machine. I am going to try to save some more RAM by going with another Turnkey Linux container template.

EDIT: In my elation over getting something to work, I forgot to change the default gateway back. Unfortunately my test machine was PORTAL, which happens to control my dynamic DNS. So currently all of my hostnames are pointed at Sweden, LÖL.

Remote Access Shenannegans

A while back, I wrote about using Windows HyperV server. The reason that I set up this server was to use the combination of a Linux server and a Windows desktop to get remote access to my home network. I thought that I would elaborate on the tools that I use to get into my home network from work or while traveling.

I use several methods, each with certain advantages and disadvantages. Mostly I prefer SSH over pretty much anything else in order to connect to a Linux host, and I prefer Remote Desktop over pretty much anything else in order to connect to a Windows host. As a backup, I will use Teamviewer. It’s not ideal, but it works where other services fail.

SSH is pretty much a Swiss Army Knife of network tools. You can use it to do waaaay more with it than just log into a Unix box and execute commands. It’s a tool for creating encrypted tunnels, it just so happens that 90% of those tunnels connect to remote shells. In addition to connecting to a remote shell, you can open ports on a host. I am fortunate enough to have Cincinnati Bell Fioptics which lets me open almost any port on my firewall without any bother. I forward port 22 directly to a Linux box named HUB, and I secure it with SSH keys. I can then use SSH to tunnel traffic into my home network, be that browser traffic through a SOCKS proxy and dynamic port, or RDP traffic with a local port. This works well when I am in a restrictive network that still allows outbound SSH traffic, and as long as I have my Putty session set up ahead of time with my private key. This is the technique that I use when I am not able to access my network through NeoRouter.

Remote Desktop (RDP) is another Swiss Army Knife for connecting to computers. I use Windows as my primary desktop OS. I like to use Linux mostly for server stuff and for running specific tools like Clonezilla or Kali. As a matter of fact, I prefer Linux for servers and tools over Windows. I know, I’m an odd duck. RDP not only gives you remote access to the Windows Desktop, it lets you map drives remotely to transfer files and it lets you connect at a desktop resolution that is greater or lesser than that of the machine that you are connecting to. This is a big deal when you are using RDP on a wide-screen monitor to control a server that is plugged into an old CRT monitor, or when you are using a tiny netbook to control your multi-screen desktop. Teamviewer (and the VNC server that it is based on) cannot do that.

In order to make my SSH and RDP connections, I like to use either NeoRouter or OpenVPN. NeoRouter is technically a split-tunneling VPN solution, but I like to think of it as creating a network of computers that is independent of their actual networks. Split-tunneling VPN is a fancy term for VPN connections that don’t mess with your Internet access. There are lots of other features for split-tunnels, but under most circumstances, I want my computers to talk to each other differently than they talk to the Internet.

The NeoRouter network explorer tool lets me see which of my computers are up and connected. I run the NeoRouter server on HUB, which is sitting behind my firewall, with port 32976 forwarded to it as well. Running the server inside my firewall lets me do some neat networking tricks, like having my BitTorrent VM connect to the internal IP for HUB, instead of using the Internet. My BitTorrent box uses a VPN client to route all Internet traffic through Sweden, which really slows down my Remote Desktop session. I run the NeoRouter client on my desktops and laptops, and also on my file servers so that I can access shared folders remotely. File transfers this way can be really slow, so I also use One Drive top share big files like videos or ISO images.

OpenVPN is my tool of choice for open WiFi networks at hotels and coffee shops. I can access my home network while also securing all of my network traffic. I run OpenVPN Access Server on a dedicated VM named GATE. Access Server is easy to use and configure, and it’s free for two concurrent connections. For occasional use, especially by people other than me, it works really well. There’s even a ready made Hyper-V appliance that you can just boot up and go. I used to run OpenVPN on HUB, but the networking/subnet stuff meant that I had to remember the internal IP for the OpenVPN network segment and change it to connect to NeoRouter. So I just use two separate machines and it all works out. I have built OpenVPN servers without Access Server in the past. I like to use the Turnkey Linux OpenVPN appliance, and setup couldn’t be easier.

If I cannot get in via NeoRouter, OpenVPN, or old school SSH tunneling, then I fall back on using TeamViewer. It can get me in when pretty much all other tools fail me, but it’s not as nice as using RDP. Also, it should be noted that TeamViewer can only be used to control graphical desktops, there is no command line equivalent. In order to alleviate some of the frustrations of TeamViewer’s desktop resolution, I run a dedicated Windows VM that I call Portal. I keep the native (console) resolution fairly low, and I have RDP and Putty sessions set up so I can quickly connect to my other computers.

One other thing that I use Portal for is to move files into and out of my home network. You can use RDP or TeamViewer to copy files, but for big files like videos and ISO’s, One Drive does a much better job. I have a dedicated One Drive account that I use specifically for moving files this way. I just grab a file from somewhere, copy it to the One Drive folder on Portal, and it automagically uploads. Then, some time later, I can use the One Drive website to download the file, at much faster speeds than using RDP, SCP (SSH), or TeamViewer’s file transfer tool. It’s an extra step, but one worth taking, especially if I find myself in an oh-shit-i-forgot-that-important-file situation.