Building a Proxmox Test Cluster in VirtualBox Part 2: Configuring the Hosts

In the last installment of this series, I discussed setting up the Proxmox VE hosts in VirtualBox. At this stage in the exercise there should be 3 VirtualBox VMs (VBVMs) running, in headless mode.

Before you can set up the cluster, storage replication, and high availability, you need to do a bit of housekeeping on your hosts. In this post, I will go over those steps making sure that the hosts are up to date OS wise, that the network interfaces are setup and communicating with eachother, and that your storage is properly configured. Most of these steps can be accomplished via the Web UI, but using SSH will be faster and more accurate. Especially when you use an SSH client like SuperPuTTY or MobaXTerm that lets you type in multiple terminals at the same time.

Log in as root@ip-address for each PVE node. In the previous post, the IPs I chose were 192.168.1.101, 192.168.1.102, and 192.168.1.103.

I don’t want to bog this post down with a bunch of Stupid SSH Tricks, so just spend a few minutes getting acquainted with MobaXTerm and thank me later. The examples below will work in a single SSH session, but you will have to paste them into 3 different windows, instead of feeling like a superhacker:

Step 1 – Fix The Subscription Thing

No, not the nag screen that pops up when you log into the web UI, the errors that you get when you try to update a PVE host with the enterprise repos enabled.

All you have to do is modify a secondary sources.list file. Open it with your editor, comment out the first line and add the second line:

nano  /etc/apt/sources.list.d/pve-enterprise.list

#deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

Save the file, and run your updates:

apt-get update; apt-get -y upgrade

While you are logged in to all 3 hosts, you might as well update the list of available Linux Container templates:

pveam update

Finally, if you set up your virtual disk files correctly according to the last post, you can set up your ZFS disk pool:

  1. List your available disks, hopefully you see two 64GB volumes that aren’t in use on /dev/sdb and /dev/sdc:
    
    root@prox1:~# lsblk
    NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                  8:0    0   32G  0 disk
    ├─sda1               8:1    0 1007K  0 part
    ├─sda2               8:2    0  512M  0 part
    └─sda3               8:3    0 31.5G  0 part
      ├─pve-swap       253:0    0  3.9G  0 lvm  [SWAP]
      ├─pve-root       253:1    0  7.8G  0 lvm  /
      ├─pve-data_tmeta 253:2    0    1G  0 lvm
      │ └─pve-data     253:4    0   14G  0 lvm
      └─pve-data_tdata 253:3    0   14G  0 lvm
        └─pve-data     253:4    0   14G  0 lvm
    sdb                  8:16   0   64G  0 disk
    sdc                  8:32   0   64G  0 disk
    sr0                 11:0    1  642M  0 rom
    root@prox1:~#
    

    Assuming you see those two disks, and they are in fact ‘sdb’ and ‘sdc’ then you can create your zpool. Which you can think of as a kind of software RAID array. There’s way more to it than that, but that’s another post for another day when I know more about ZFS. For this exercise, I wanted to make a simulated RAID1 array, for “redundancy.” Set up the drives in a pool like so:

    
    zpool create -f -o ashift=12 z-store mirror sdb sdc
    zfs set compression=lz4 z-store
    zfs create z-store/vmdata
    

    In a later post we will use the zpool on each host for Storage Replication. The PVEVM files for each of your guest machines will copy themselves to the other hosts at regular intervals so when you migrate a guest from one node to another it won’t take long. This feature pairs very well with High Availability, where your cluster can determine if a node is down and spin up PVEVMs that are offline.

    Now that your disks are configured, it’s time to move on to Part 3: Building A Cluster Network.

    Advertisements

Building a Proxmox Test Cluster in VirtualBox Part 1: Building The Hosts

In my last post, I set the stage for why I built the virtualbox cluster, and now it it time to discuss the how.

In researching the best way to design a network for a Proxmox cluster, the bare minimum is one network connection. This one link does the following:

  1. Hosts the web server for the management GUI – The web UI is pretty slick, and it’s great for viewing stats and checking for errors.
  2. Hosts the network bridge for guest VMs – This bridge acts as a kind of virtual network switch for your PVEVMs to talk to the outside world.
  3. Connects the host to the Internet – The PVE host needs to download security updates, Linux container templates, and install packages.

This one network interface is sort of the lifeline for a Proxmox host. It would be a shame if that link got bombed by incessant network traffic. As I discovered (the hard way) one possible source of incessant network traffic is the cluster communication heartbeat. Obviously, that traffic needs to go on its own network segment. Normally, that would be a VLAN or something, but I have some little dumb switches and the nodes have some old quad port NICs, so I wanted to just assign an IP to one port, and plug that port into a switch that is physically isolated from “my” network.

Once a cluster is working, migrating machines happens over the cluster network link. This is OK, but if your cluster network happens to suck (like when some jackass plugs it into a 10 year old switch) it can cause problems with determining if all the cluster nodes are online. So, now I want to set up an additional interface for VM migration. Migration seems like the kind of thing that happens only occasionally, but when you enable Storage Replication, the nodes are copying data every 15 minutes. Constant cluster chatter, plus constant file synchronization, has the potential to saturate a single network link. This gets even worse when you add High Availability, and there is a constant vote on if a PVEVM is up and running, followed by a scramble to get it going on another node.

So, at minimum we will need 3 network interfaces for the test cluster on VirtualBox. I didn’t want to spend a lot of time tinkering with firewall and NAS appliances, so I am leaving the “Prox management on its own network segment” and the “Dedicated network storage segment” discussions out of this exercise. I can’t decide if the management interface for my physical Proxmox cluster should sit on my internal network, or on its own segment. For this exercise, the management interface is going to sit on the internal network. My Synology NAS has 4 network ports, so I am definitely going to dedicate a network segment for the cluster to talk to the NAS, but that won’t be a part of this exercise.

[Virtual] Hardware Mode(tm)

Once you are booted up and VirtualBox is running, you can start building your VBVMs. I recommend building one VBVM to use as a template and then cloning it 3 times. I found that I kept missing important things and having to start over, so better to fix the master and then destroy the clones.

I called my master image “proxZZ” so it showed up last in the list of VBVMs. I also never actually started up the master image, so it was always powered off and the ZZ’s made it look like it was sleeping.

Create proxZZ with the following:

  • First, make sure that you have created 2 additional Host Only Network Adapters in VirtualBox. In this exercise you will only use two, but it can get confusing when you are trying to match en0s9 to something, so do yourself a favor and make three. Make sure to disable the DHCP server on both adapters.
  • Create a new virtual machine with the following characteristics :
    1. Name: ProxZZ
    2. Type: Linux
    3. Version: Debian 64bit (Proxmox is Debian under the hood.)
    4. Memory Size: 2048MB
    5. Hard drive: dynamically allocated, 32GB in size.
  • Make sure that you have created 3 total virtual hard disks as follows:
    1. SATA0: 32GB. This will be your boot drive and system disk. This is where Proxmox PVE will be installed. Static disks are supposed to be faster, but this isn’t even remotely about speed. My laptop has a 240gb SSD, so I don’t have a ton of space to waste.
    2. SATA1: 64GB, dynamically allocated. This will be one of your ZFS volumes.
    3. SATA2: 64GB, dynamically allocated. This will be your other ZFS volume. Together they will make a RAID1 array.
  • WHile you are in the storage tab, make sure to mount the Proxmox installer ISO
  • Make sure that you have created 3 network interfaces as follows:
    1. Adapter 1: Bridged Network – this will be your management interface.
    2. Adapter 2: Host Only Network Adapter #2 – this will be your cluster interface.
    3. Adapter 3: Host Only Network Adapter #3 – this will be your VM migration interface.
    4. You may be tempted to do something clever like unplugging virtual cables or something. Don’t. You will be cloning this machine in a minute and you will have a hard time keeping all of this straight.
  • Before you finish, make sure that the machine is set to boot from the hard drive first, followed by the CD/Optical drive. This seems stupid, but you will be booting these things in headless mode, and forgetting to eject the virtual CD rom is super annoying. So fix it here and stop being bothered with it.

When it’s done, it should look something like this:

Once you are sure your source VM is in good shape, make 3 clones of it. Don’t install Proxmox yet. SSH keys and stuff will play a major role in this exercise later, and I am not sure if VirtualBox is smart enough to re-create them when you clone it. I ran into this a few times so just clone the powered off VBVM. I called the clones prox1, prox2, and prox3.

[Virtual] Software Mode(tm)

Now it is time to start your 3 clones. This can get pretty repetitive, especially if you start the process over a couple of times. While you will appreciate cloning the servers, there isn’t really a simple way that I have discovered to build the PVE hosts. In a few iterations of this exercise, I misnamed one of the nodes (like pro1 or prx2) and it’s super annoying later when you get the cluster set up and see one of the nodes named wrong. There is a procedure to fix the node name after you build it, but seriously just take your time and pay attention.

As you do the install, select your 32gb boot drive and configure your IP addresses.
I went with a sequence based on the hostname:
prox1 – 192.168.1.101
prox2 – 192.168.1.102
prox3 – 192.168.1.103
Like I said before, go slowly and pay attention. This part is super repetitive and it’s easy to make a stupid mistake that you have to troubleshoot later. At some point, I guarantee that you will give up, destroy the clones, and start over 🙂

Send In The Clones

Once your hosts are installed, it’s time to shut them down and boot them again, this time in headless mode. This is where fixing the boot order on ProxZZ pays off. With all 3 VBVMs are started up, you are ready for the next stage of the exercise: configuring your hosts.

Turnkey Torrents and Swedish Internets

A few months ago, I wrote about using a Turnkey Linux OpenVPN appliance to route network traffic thru Sweden. Since that time I have gotten my BitTorrent machine running. The other post was mostly about getting the VPN tunneling and routing to work. This post will mostly be about setting up the torrent server.

The Turnkey Torrent Server is neat because it’s a minimal Debian machine with a pre-configured Transmission BitTorrent Client, a web interface for managing BitTorrent, a working Samba server, and a webDAV client so you can use a browser to download files. Basically, you use the web interface to grab things, the Samba server to makes them accessible to your media players on your internal network, and webDAV makes the files accessible to the rest of the world, assuming you have the right ports forwarded. My preferred method for watching torrented videos is on a PC plugged into a TV running VLC Media player controlled with a wireless keyboard. I know I should be using Plex and shit like that, but I’m old school.

The Swedish Connection
For some of my friends who aren’t pirates (especially the friends that are into British TV) I am like their coke dealer except I deal in movies and TV shows. That means that sometimes I get asked to find things when I’m not at home. Like a third of my remote access shenanigans, A.K.A. reverse telecommuting, is so that I can pull up BitTorrent and snarf shit for friends and relatives when I’m not at home. Being able to expose the uTorrent remote interface to the web was great for letting my more technical non-hacker friends grab torrents without any assistance from me.

My VPN provider gives me the option of forwarding ports. When I was running uTorrent on a dedicated Windows machine, those forwarded ports were easy to configure. I would just set them up on the VPN site and map them to the ports I configured in uTorrent. One was for BitTorrent transfers to make sure that my ratios reported correctly on private trackers. The other was for the uTorrent web interface. For a long time I ran Windows for torrenting because I used PeerBlock to help me fly under the radar. Times change tho. Real time block lists is old and busted. VPNs is the new hotness. Unfortunately, with this VPN router setup it messes up forwarding ports. When I set up port forwarding on the VPN provider side, the forwarded ports hit the doorway server rather than the torrent server, so that has to be fixed with more IPTables kung fu on the doorway server.

I know I said that I wasn’t going to write anymore about the doorway server, but I lied. I needed to configure the doorway server to open those ports and then forward them to the torrent server. Let’s assume that my internal network is a 192.168.1.0/24 subnet (a class A block, a range of addresses from 192.168.1.1 to 192.168.0.254) with a default gateway of 192.168.1.1. All of my traffic goes through my local router and hits the Internet from my ISP, in the US. If a device asks for an IP via DHCP, this is the network configuration that it will receive, along with red-blooded American Internets. Here is an awful network diagram because why not?

The doorway server has a static IP of 192.168.1.254 and it’s configured to route all of its traffic through the VPN tunnel to Sweden. Any device that is configured to have a default gateway of 192.168.1.254 will also hit the Internet via the tunnel to Sweden, thereby receiving Swedish Internets. At this point, all the configuration is done, and your torrent server will work, but there won’t be any ports forwarded to it, which is lame. No forwarded ports is especially lame when you are using private trackers because it can really mess with your ratios. Now, you could just open a port on your firewall for the web interface on the American side, but that’s also pretty lame. If you want to use your torrent server, you should strictly be using Swedish Internets.

Welcome to Swedish Internet
To forward those ports, first set them up in Transmission, then with your VPN provider. The web interface port [12322] is already configured for you by Turnkey Linux. You can set the other port in the Preferences->Network->Listening Port blank. Once the entry points and the end points are configured, it’s time to do more iptables kung fu.

Let’s assume the following:

  1. The web interface port for Transmission is 12322.
  2. The listening port in Transmission to 9001.
  3. The static IP for your torrent server is 192.168.1.10
  4. The doorway server IP is 192.168.1.254.
  5. The forwarding ports you were able to get from your VPN provider are 9000 and 9001.
  6. You want to use port 9000 on the VPN side for the Transmission web interface.
  7. You wand to use port 9001 on the VPN side for the Transmission listening port.

What needs to happen is for the VPN tunnel interface (tun0) to listen on ports 9000 and 9001, then forward traffic on those ports to 192.168.1.10. Then, you want any traffic on those same ports that comes from the doorway’s internal network interface (eth0) to be modified so that it doesn’t look like it came from the tunnel interface. This is super important for TCP handshakes.

First create your rules for accepting/forwarding connections on the VPN side:


iptables -A FORWARD -i tun0 -o eth0 -p tcp --syn --dport 9000 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -i tun0 -o eth0 -p udp --dport 9001 -m conntrack --ctstate NEW -j ACCEPT

This was probably configured fine in the doorway server post, but this specifically allows all the traffic that passes between your VPN and the local network connections once a connection is establshed:


iptables -A FORWARD -i eth0 -o tun0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i tun0 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Now add the rules to rewrite packets destined to the web interface and then rewrite the responses:


iptables -t nat -A PREROUTING -i tun0 -p tcp --dport 9000 -j DNAT --to-destination 192.168.1.10:12322
iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 9000 -d 192.168.1.10 -j SNAT --to-source 192.168.1.254

Add the rules to rewrite all the BitTorrent packets, including responses:


iptables -t nat -A PREROUTING -i tun0 -p udp --dport 9001 -j DNAT --to-destination 192.168.1.10:9001
iptables -t nat -A POSTROUTING -o eth0 -p udp --dport 9001 -d 192.168.38.37 -j SNAT --to-source 192.168.1.254

All the strict rewriting probably isn’t a big deal for the BitTorrent traffic because it’s UDP, and UDP don’t give a fuck.

If it’s working, point your browser to https://the-ip-address-of-your-vpn-server:9000 and you should be prompted to log in to the web interface. Once you’re sure it’s all good, then it’s time to save your working iptables config:

iptables-save | tee /etc/iptables.up.rules

Make sure that your rules work well after you reboot your VM. And then run your backups to make sure that they have your latest config because there’s nothing worse than trying to piece all this crap together for the third time.

You can skip having to remember the IP by registering it as a subdomain somewhere, either with a dynamic DNS service, or with the registrar for a domain that you own.

In the unlikely event that I made this, or any other technical thing look easy, rest assured that it took me at least a couple hours. Also, I had it working a months ago, but I forgot to update my snapshot and had to redo it again because I am not a smart man. Then during this second go around I had to restore the VM from a backup because iptables just isn’t my bag. Thankfully BitTorrent is my bag. Happy pirating!

Adventures in Proxmox Part 1: Words About Boxes

The Proxmox logo
It’s been a few weeks since I exorcised HyperV from my life like an evil demon. I have replaced it with Proxmox and so far it’s been mostly great. With a couple of serious caveats.

Plastic dinosaurs betraying each other.My transition to Proxmox has been a rather involved, not so much because Proxmox is hard to set up (it’s not), but because I am tired of slapping old junky hardware together and hoping it doesn’t die, and then scrambling to fix it when it inevitably betrays me. Unlike most dudes with home servers and labs, most of my acquisitions were made years ago to support an MMO habit. Specifically multiboxing.

PC case made from peg board.

I call them “computers” because they are computers in the sense that they have CPU’s, RAM, and HDD’s. But they were low-budget things when they were assembled years ago. The upgrade path works something like this:

  1. A computer begins its life as my main gaming machine that will run my favorite game at a satisfactory speed and resolution.
  2. Then I find a new favorite and upgrade the gaming machine’s guts to run the new game.
  3. The old gaming guts get transplanted in to my “server” where they are *barely* able to run a few VMs and things like that.
  4. The final stage is when the server guts are no longer up to the task of running VMs. I then add a few old network cards and the “server” then becomes my “router”.
  5. The old router guts then get donated somewhere. They’re not really useful to anyone, so they probably get shipped to Africa where they get mined for gold and copper by children at gunpoint.

Breaking the [Re]Cycle of Violence
Wall-E holding a pile of scrapIn the years since then, I have taken to playing epic single player games like Skyrim. These games really only need one machine. The rest of the gear I used to run little “servers” for one thing or another, which I have slowly replaced with VMs. The problem with using old junky computers as servers is when you run them balls out 24 hours a day. In my search for a replacement VM host, I spent a lot of time researching off-lease servers. My goal was to have 8 cores and 32gb of ram, with the ability to live migrate VMs to another [lesser] host in an emergency, something that my HyperV setup was lacking. After a lot of consternation, I decided that since a single VM would never actually use more than 4 cores or 8gb of RAM, why not use 2 [or more] desktops?

A room full of old PCs.I found some old off-lease quad-core Intel desktops for about the same retail price as a low end server processor. I used the RAM from my older gaming machines/VMservers and some hard drives from some old file servers to build out my “new” Proxmox cluster. With two quad core desktops running maxed-out memory(16GB each) I managed to satisfy my need to be like the other kids with “8 cores with 32GB of RAM” for about the price of an off-lease server chassis, with the added bonus having a cluster. The goal is to add nodes to grow the cluster to 16 cores and 64GB of RAM, while also adding clustered storage via Ceph to make use of old hard drives from file servers.

New hot servers is old and busted. Old busted clusters is the new hotness.
For me, the clustered model is better, in my opinion for a number of reasons. It mostly has to do with modularity:

  1. You can build out your infrastructure one paycheck at a time. Part of the problem with off-lease servers is that while the chassis is cheap, the components that go in it are expensive and/or hard to find. The deal with servers is that the cost of the motherboard and CPU are nothing compared to what you will spend on RAM. I was looking for something I could start using for less than $200, and a refurb desktop and RAM from old gaming boxes got me going at that price point.
  2. Desktops stack on top of each other for free. I don’t have any server or telco racks, so in addition buying ECC RAM, I would also be buying a rack, rails, and all of the other stuff that goes with them. This would easily eat up my $200 startup budget before I powered on a single box.
  3. Moar boxes == moar resiliency. My gear at home is part lab and part production environment. Yes, I use it to hack stuff and learn new things, but my family also uses it in their daily lives. Network shares stream cartoons; VOIP phones connect friends; keeping these things going is probably as important as my day job. Being able to try bold and stupid things without endangering the “Family Infrastructure” is important to my quality of life.
  4. Scaling out is probably more important than Scaling Up. A typical I.T. Department/Data Center response to capacity problems is to regularly stand up newer/more powerful [expensive] gear and then dump the old stuff. I guess this is a good approach if you have the budget. It certainly has created a market for used gear. I don’t have any budget to speak of, so I want to be able to increase capacity by adding servers while keeping the existing ones in play. There are still cost concerns with this approach, mainly with network equipment. In addition to upping my server game, I am going to have to up my networking game as well.

It works…ish

I have my two cluster nodes *kind of* working, with most of my Linux guests running as containers, which is very memory and CPU efficient. I am running two Windows VMs, PORTAL for remote access and dynamic DNS, and MOONBASE which I am using for tasks that need wired network access. All of my desktops are currently in pieces, having donated their guts to the “Cluster Collective” so I am mostly using my laptop for everything. I am not really in the habit of plugging it in to Ethernet, or leaving it turned on, so for now I am using a VM in place of my desktop for long running tasks like file transfers.

I say that the cluster is only kind of working because my home network isn’t very well segmented and the cluster heartbeat traffic straight up murders my little switch. It took me a while to figure out the problem. So the cluster works for a few days and then my core switch chokes and passes out, knocking pretty much everything offline. For now, the “cluster” is disabled and the second node is powered off until my new network cards arrive and I can configure separate networks for the clustering, storage, and the VMs.

Coming soon: Adventures in Proxmox part 2: You don’t know shit about networking.