Building a Proxmox Test Cluster in VirtualBox Part 4: Containers, Storage, and Replication

In the last installment in this series, I showed you how to build a cluster with separate interfaces dedicated to the cluster heartbeat traffic and to virtual machine migration. In order to see these in action, you need for your PVE hosts to run some VMs of their own. Once there are a three VMs running, we can play with the really great features of a Proxmox Cluster, like storage replication and high availability.

During the initial setup of the Proxmox hosts on VirtualBox, you probably saw an error message about KVM virtualization. I dismissed it as not a big deal. The truth is that this whole “virtualization inside of virtualization” exercise won’t produce any useful PVEVMs. I haven’t done much troubleshooting, but I am fairly certain that while the PVEVMs boot up and you can log into their consoles, they don’t really talk to each other or your physical internal network. That’s not as cool as I had hoped, but the point of this exercise was to figure out the network design for a Proxmox Cluster, not to play with cool double-layer virtual machines. I am sure that you can so some cool router Kung Fu to get them going for real, but for this exercise, PVEVMs booting up is enough. We’re just going to move them around the cluster so see how it all works.

Also, because each PVE cluster node is woefully under-powered, RAM is really at a premium:

So even if they did work as advertised, the PVEVM’s probably wouldn’t work well. Linux Containers are awesomely efficient, especially with RAM, but they can’t squeeze blood from a stone.

Downloading Templates and Building Containers

To build your PVEVMs you need to download a container template. It can be any of them. I prefer the LXC Debian 9 template or the Turnkey Linux Core template. How you want to go about building the test PVEVMs is your business, but the goal is 3 PVEVMs up and running. At different points in this exercise there will be one PVEVM running on each PVE cluster node, and all 3 PVEVMs running on one node. It’s up to you if you want to build them out:

  • Download the template to one node, build the container, and then migrate it to another node, OR
  • Download the template to each node and build each PVEVM from there, OR
  • Download the template, build the container, and then clone it.
  • The process is low as hell no matter how you slice it.

Once you have decided your approach, build 3 new containers. It takes a long time, but it demonstrates your need for a central data store for non-VM files. This is where a NAS would be handy. You could set up the NAS to store ISOs, container templates, and backups so that they would be accessible to all the nodes.

It’s important to note that you should build privileged containers. You do this by UN-checking the UN-privileged box. It’s stupid; I know.

At this point, you can migrate a container from one node to the other, but it takes a long time because you have to wait for the container to completely shut down, and then for the container files to completely copy from one cluster node to the next. On real hardware, this process will probably go a bit faster, but this is a good illustration of why we need storage replication.

Storage Replication

Before we can enable replication, we need to set up the ZFS storage properly. In the Building the Cluster Nodes post, we set up a ZFS array called ZStore, and now it’s time to set it up for the whole cluster.

In the Storage View for the cluster you have the option of adding storage. Here you will add a ZFS type and include the zstore/vmdata. Make sure to add all 3 nodes. I called mine “ZVMZ.” At this point it should be apparent that a RAID1 mirror, that is going to be replicated to 2 other mirrors is probably overkill in terms of redundancy, so you should probably do something different on your real hardware. If you have small but fast disks, you might RAID0 them to get nice write speeds, then replicate them for redundancy. Or do whatever.

Once the ZFS storage is set up for all of your nodes, you can move the storage for your PVEVMs to the ZVMZ storage. This will take a long time as well because everything has to copy over. This should be the last time you have to sit through a full copy of anything. Now we can set up replication.

Replication is done on a per VM and per host basis. So you will want to make sure that each VM has a job created to replicate to the other nodes. You only need two jobs for each VM. If you migrate a VM to another host, the replication job will update. The first time the job runs it will take a while. There isn’t a progress bar or anything, so you will have to check the ZVMZ storage on each node to make sure that there are copies of all of your VMs.

With replication set up, even if a cluster node fails, you will only lose 15 minutes (assuming you went with the default schedule) worth of data on the server and you can start up your server on another node with the snapshot. Migrate some VMs and see for yourself, and stay tuned for the next installment: High Availability.

Building a Proxmox Test Cluster in VirtualBox Part 3: Building The Cluster

In the last installment of this series, I discussed setting up the Proxmox VE hosts. Until now, you could do most of this configuring in triplicate with MobaXTerm. Now you can still use it to multicast, just be sure to disable it when you have to customize the configs for each host. This part of the process is a lethal combination of being really repetitive while also requiring a lot of attention to detail. This is also the point where it gets a bit like virtualization-inception: VirtualBox VMs which are PVE hosts to PVEVMs.

Network Adapter Configuration
I did my best to simplify the network design:

  • There are 3 PVE hosts with corresponding management IP’s:
    1. prox1 – 192.168.1.101
    2. prox2 – 192.168.1.102
    3. prox3 – 192.168.1.103
  • Each PVE host has 3 network adapters:
    1. Adapter 1: A Bridged Adapter that connects to the [physical] internal network.
    2. Adapter 2: Host only Adapter #2 that will serve as the [virtual] isolated cluster network.
    3. Adapter 3: Host only Adapter #3 that will serve as the [virtual] dedicated migration network.
  • Each network adapter plugs into a different [virtual] network segment with a different ip range:
    1. Adapter 1 (enp0s3) – 192.168.1.0/24
    2. Adapter 2 (enp0s8) – 192.168.2.0/24
    3. Adapter 3 (enp0s9) – 192.168.3.0/24
  • Each PVE hosts’ IP on each network roughly corresponds to its hostname:
    1. prox1 – 192.168.1.101, 192.168.2.1, 192.168.3.1
    2. prox2 – 192.168.1.102, 192.168.2.2, 192.168.3.2
    3. prox3 – 192.168.1.103, 192.168.2.3, 192.168.3.3

I have built this cluster a few times and my Ethernet adapter names (enp0s3, enp0s8, and enp0s9) have always been the same. That may be a product of all the cloning, so YMMV. Pay close attention here because this can get very confusing.

Open the network interface config file for each PVE host:

nano /etc/network/interfaces

You should see the entry for your first Ethernet adapter (the bridged adapter in VirtualBox), followed by the virtual machines' bridge interface with the static IP that you set when you installed Proxmox. This is a Proxmox virtual Ethernet adapter. The last two entries should be your two host only adapters, #2 and #3 in VirtualBox. These are the adapters that we need to modify. The file for prox1 probably looks like this:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

iface enp0s3 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.101
        netmask 255.255.255.0
        gateway 192.168.1.1
        bridge_ports enp0s3
        bridge_stp off
        bridge_fd 0

iface enp0s8 inet manual

iface enp0s9 inet manual

Ignore the entries for lo, enp0s3, and vmbr0. Delete the last two entries (enp0s8 and enp0s9) and replace them with this:

#cluster network
auto enp0s8
iface enp0s8 inet static
        address 192.168.2.1
        netmask 255.255.255.0
#migration network
auto enp0s9
iface enp0s9 inet static
        address 192.168.3.1
        netmask 255.255.255.0

Now repeat the process for prox2 and prox3, changing the last octet for the IP's to .2 and .3 respectively. When you are finished your nodes should be configured like so:

  1. prox1 - 192.168.1.101, 192.168.2.1, 192.168.3.1
  2. prox2 - 192.168.1.102, 192.168.2.2, 192.168.3.2
  3. prox3 - 192.168.1.103, 192.168.2.3, 192.168.3.3

Save your changes on each node. Then reboot each one:

shutdown -r now

Network Testing
When your PVE hosts are booted back up, SSH into them again. Have each host ping every other host IP to make sure everything is working:

ping -c 4 192.168.2.1;\
ping -c 4 192.168.2.2;\
ping -c 4 192.168.2.3;\
ping -c 4 192.168.3.1;\
ping -c 4 192.168.3.2;\
ping -c 4 192.168.3.3

The result should be 4 replies from each IP on each host with no packet loss. I am aware that each host is pinging itself twice. But you have to admit it look pretty bad ass.

Once you can hit all of your IP's successfully, now it's time to make sure that multicast is working properly. This isn't a big deal in VirtualBox because the virtual switches are configured to handle multicast correctly, but it's important to see the test run so you can do it on real hardware in the future.

First send a bunch of multicast traffic at once:

omping -c 10000 -i 0.001 -F -q 192.168.2.1 192.168.2.2 192.168.2.3

You should see a result that is 10000 sent with 0% loss. Like so:

192.168.2.1 :   unicast, xmt/rcv/%loss = 9406/9395/0%, min/avg/max/std-dev = 0.085/0.980/15.200/1.940
192.168.2.1 : multicast, xmt/rcv/%loss = 9406/9395/0%, min/avg/max/std-dev = 0.172/1.100/15.975/1.975
192.168.2.2 :   unicast, xmt/rcv/%loss = 10000/9991/0%, min/avg/max/std-dev = 0.091/1.669/40.480/3.777
192.168.2.2 : multicast, xmt/rcv/%loss = 10000/9991/0%, min/avg/max/std-dev = 0.173/1.802/40.590/3.794

Then send a sustained stream of multicast traffic for a few minutes:

omping -c 600 -i 1 -q 192.168.2.1 192.168.2.2 192.168.2.3

Let this test run for a few minutes. Then cancel it with CTRL+C.

The result should again be 0% loss, like so:

root@prox1:~# omping -c 600 -i 1 -q 192.168.2.1 192.168.2.2 192.168.2.3
192.168.2.2 : waiting for response msg
192.168.2.3 : waiting for response msg
192.168.2.3 : joined (S,G) = (*, 232.43.211.234), pinging
192.168.2.2 : joined (S,G) = (*, 232.43.211.234), pinging
^C
192.168.2.2 :   unicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.236/1.488/6.552/1.000
192.168.2.2 : multicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.338/2.022/7.157/1.198
192.168.2.3 :   unicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.168/1.292/7.576/0.905
192.168.2.3 : multicast, xmt/rcv/%loss = 208/208/0%, min/avg/max/std-dev = 0.301/1.791/8.044/1.092

Now that your cluster network is up and running, you can finally build your cluster. Up to this point, you have been entering identical commands into a SSH sessions. At this point, you can stop using the multi-exec feature of your SSH client.

First, create the initial cluster node on Prox1, like so:

root@prox1:~# pvecm create TestCluster --ring0_addr 192.168.2.1 --bindnet0_addr 192.168.2.0

Then join Prox2 to the new cluster:

root@prox2:~# pvecm add 192.168.2.1 --ring0_addr 192.168.2.2

Followed by Prox3:

root@prox3:~# pvecm add 192.168.2.1 --ring0_addr 192.168.2.3

One final configuration on Prox1 is to set the third network interface as the dedicated migration network by updating the datacenter.cfg file, like so:

root@prox1:~# nano /etc/pve/datacenter.cfg

keyboard: en-us
migration: secure,network=192.168.3.0/24

Now that the cluster is set up, you can log out of your SSH sessions and switch to the web GUI. When you open the web GUI for Prox1 (https://192.168.1.101:8006) and you should see all 3 nodes in your TestCluster:

Now you can manage all of your PVE Hosts from one graphical interface. You can also do cool shit like migrating VMs from one host to another, but before we can do that we need to set up some PVEVMs. There are more things to set up, but to see them in action, we need to build a couple of PVEVMS to work with, which I will cover in the next installment: Building PVE Containers.

Building a Proxmox Test Cluster in VirtualBox Part 2: Configuring the Hosts

In the last installment of this series, I discussed setting up the Proxmox VE hosts in VirtualBox. At this stage in the exercise there should be 3 VirtualBox VMs (VBVMs) running, in headless mode. These VBVMs should be running Proxmox VE, ready to host their own VMs (PVEVMs).

Before you can set up the cluster, storage replication, and high availability, you need to do a bit of housekeeping on your hosts. In this post, I will go over those steps making sure that the hosts are up to date OS wise and that your storage is properly configured for later use. Most of these steps can be accomplished via the Web UI, but using SSH will be faster and more accurate. Especially when you use an SSH client like SuperPuTTY or MobaXTerm that lets you type and paste into multiple terminals at the same time.

Log in as root@ip-address for each PVE node. In the previous post, the IPs I chose were 192.168.1.101, 192.168.1.102, and 192.168.1.103.

I don’t want to bog this post down with a bunch of Stupid SSH Tricks, so just spend a few minutes getting acquainted with MobaXTerm and thank me later. The examples below will work in a single SSH session, but you will have to paste them into 3 different windows like a peasant, instead of feeling like some kind of superhacker:

Step 1 – Fix The Subscription Thing

No, not the nag screen that pops up when you log into the web UI, I mean the errors that you get when you try to update a PVE host with the enterprise repos enabled.

All you have to do is modify a secondary sources.list file. Open it with your editor, comment out the first line and add the second line:

nano  /etc/apt/sources.list.d/pve-enterprise.list

# old and busted
#deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

# new hotness
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

Save the file, and run your updates:

apt-get update; apt-get -y upgrade

While you are logged in to all 3 hosts, you might as well update the list of available Linux Container templates:

pveam update

Finally, if you set up your virtual disk files correctly according to the last post, you can set up your ZFS disk pool:

  1. List your available disks, hopefully you see two 64GB volumes that aren’t in use on /dev/sdb and /dev/sdc:
    
    root@prox1:~# lsblk
    NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                  8:0    0   32G  0 disk
    ├─sda1               8:1    0 1007K  0 part
    ├─sda2               8:2    0  512M  0 part
    └─sda3               8:3    0 31.5G  0 part
      ├─pve-swap       253:0    0  3.9G  0 lvm  [SWAP]
      ├─pve-root       253:1    0  7.8G  0 lvm  /
      ├─pve-data_tmeta 253:2    0    1G  0 lvm
      │ └─pve-data     253:4    0   14G  0 lvm
      └─pve-data_tdata 253:3    0   14G  0 lvm
        └─pve-data     253:4    0   14G  0 lvm
    sdb                  8:16   0   64G  0 disk
    sdc                  8:32   0   64G  0 disk
    sr0                 11:0    1  642M  0 rom
    root@prox1:~#
    

    Assuming you see those two disks, and they are in fact ‘sdb’ and ‘sdc’ then you can create your zpool. Which you can think of as a kind of software RAID array. There’s waaaay more to it than that, but that’s another post for another day when I know more about ZFS. For this exercise, I wanted to make a simulated RAID1 array, for “redundancy.” Set up the drives in a pool like so:

    
    zpool create -f -o ashift=12 z-store mirror sdb sdc
    zfs set compression=lz4 z-store
    zfs create z-store/vmdata
    

    In a later post we will use the zpool on each host for Storage Replication. The PVEVM files for each of your guest machines will copy themselves to the other hosts at regular intervals so when you migrate a guest from one node to another it won’t take long. This feature pairs very well with High Availability, where your cluster can determine if a node is down and spin up PVEVMs that are offline.

    Now that your disks are configured, it’s time to move on to Part 3: Building The Cluster.

Building a Proxmox Test Cluster in VirtualBox Part 1: Building The Hosts

In my last post, I set the stage for why I built the virtualbox cluster, and now it it time to discuss the how.

In researching the best way to design a network for a Proxmox cluster, the bare minimum is one network connection. This one link does the following:

  1. Hosts the web server for the management GUI – The web UI is pretty slick, and it’s great for viewing stats and checking for errors.
  2. Hosts the network bridge for guest VMs – This bridge acts as a kind of virtual network switch for your PVEVMs to talk to the outside world.
  3. Connects the host to the Internet – The PVE host needs to download security updates, Linux container templates, and install packages.

This one network interface is sort of the lifeline for a Proxmox host. It would be a shame if that link got bombed by incessant network traffic. As I discovered (the hard way) one possible source of incessant network traffic is the cluster communication heartbeat. Obviously, that traffic needs to go on its own network segment. Normally, that would be a VLAN or something, but I have some little dumb switches and the nodes have some old quad port NICs, so I wanted to just assign an IP to one port, and plug that port into a switch that is physically isolated from “my” network.

Once a cluster is working, migrating machines happens over the cluster network link. This is OK, but if your cluster network happens to suck (like when some jackass plugs it into a 10 year old switch) it can cause problems with determining if all the cluster nodes are online. So, now I want to set up an additional interface for VM migration. Migration seems like the kind of thing that happens only occasionally, but when you enable Storage Replication, the nodes are copying data every 15 minutes. Constant cluster chatter, plus constant file synchronization, has the potential to saturate a single network link. This gets even worse when you add High Availability, and there is a constant vote on if a PVEVM is up and running, followed by a scramble to get it going on another node.

So, at minimum we will need 3 network interfaces for the test cluster on VirtualBox. I didn’t want to spend a lot of time tinkering with firewall and NAS appliances, so I am leaving the “Prox management on its own network segment” and the “Dedicated network storage segment” discussions out of this exercise. I can’t decide if the management interface for my physical Proxmox cluster should sit on my internal network, or on its own segment. For this exercise, the management interface is going to sit on the internal network. My Synology NAS has 4 network ports, so I am definitely going to dedicate a network segment for the cluster to talk to the NAS, but that won’t be a part of this exercise.

[Virtual] Hardware Mode(tm)

Once you are booted up and VirtualBox is running, you can start building your VBVMs. I recommend building one VBVM to use as a template and then cloning it 3 times. I found that I kept missing important things and having to start over, so better to fix the master and then destroy the clones.

I called my master image “proxZZ” so it showed up last in the list of VBVMs. I also never actually started up the master image, so it was always powered off and the ZZ’s made it look like it was sleeping.

Create proxZZ with the following:

  • First, make sure that you have created 2 additional Host Only Network Adapters in VirtualBox. In this exercise you will only use two, but it can get confusing when you are trying to match en0s9 to something, so do yourself a favor and make three. Make sure to disable the DHCP server on both adapters.
  • Create a new virtual machine with the following characteristics :
    1. Name: ProxZZ
    2. Type: Linux
    3. Version: Debian 64bit (Proxmox is Debian under the hood.)
    4. Memory Size: 2048MB
    5. Hard drive: dynamically allocated, 32GB in size.
  • Make sure that you have created 3 total virtual hard disks as follows:
    1. SATA0: 32GB. This will be your boot drive and system disk. This is where Proxmox PVE will be installed. Static disks are supposed to be faster, but this isn’t even remotely about speed. My laptop has a 240gb SSD, so I don’t have a ton of space to waste.
    2. SATA1: 64GB, dynamically allocated. This will be one of your ZFS volumes.
    3. SATA2: 64GB, dynamically allocated. This will be your other ZFS volume. Together they will make a RAID1 array.
  • WHile you are in the storage tab, make sure to mount the Proxmox installer ISO
  • Make sure that you have created 3 network interfaces as follows:
    1. Adapter 1: Bridged Network – this will be your management interface.
    2. Adapter 2: Host Only Network Adapter #2 – this will be your cluster interface.
    3. Adapter 3: Host Only Network Adapter #3 – this will be your VM migration interface.
    4. You may be tempted to do something clever like unplugging virtual cables or something. Don’t. You will be cloning this machine in a minute and you will have a hard time keeping all of this straight.
  • Before you finish, make sure that the machine is set to boot from the hard drive first, followed by the CD/Optical drive. This seems stupid, but you will be booting these things in headless mode, and forgetting to eject the virtual CD rom is super annoying. So fix it here and stop being bothered with it.

When it’s done, it should look something like this:

Once you are sure your source VM is in good shape, make 3 clones of it. Don’t install Proxmox yet. SSH keys and stuff will play a major role in this exercise later, and I am not sure if VirtualBox is smart enough to re-create them when you clone it. I ran into this a few times so just clone the powered off VBVM. I called the clones prox1, prox2, and prox3.

[Virtual] Software Mode(tm)

Now it is time to start your 3 clones. This can get pretty repetitive, especially if you start the process over a couple of times. While you will appreciate cloning the servers, there isn’t really a simple way that I have discovered to mass produce the PVE hosts. In a few iterations of this exercise, I misnamed one of the nodes (like pro1 or prx2) and it’s super annoying later when you get the cluster set up and see one of the nodes named wrong. There is a procedure to fix the node name after you build it, but seriously just take your time and pay attention.

When you first start the install you will probably see an error about KVM Virtualization not being detected. Which should be impossible, right? The PVE host is *literally* a virtual machine. This is probably VBox for Windows not passing something to PVE Linux, or VBox misidentifying your CPU. Whatever it is, this isn’t a showstopper because our test systems that will run on the PVE cluster aren’t going to be VMs, they’ll be Linux Containers, but in the interest of not making my head hurt I am still going to call them VMs. The PVE hosts are VirtualBox VMs, and the Linux Containers running on Proxmox are Proxmox VE VMs. VBVMs are VMs running under VirtualBox, PVEVMs are Linux Containers running under Proxmox VE. *whew*

As you do the install, select your 32gb boot drive and configure your IP addresses.
I went with a sequence based on the hostname:
prox1 – 192.168.1.101
prox2 – 192.168.1.102
prox3 – 192.168.1.103
Like I said before, go slowly and pay attention. This part is super repetitive and it’s easy to make a stupid mistake that you have to troubleshoot later. At some point, I guarantee that you will give up, destroy the clones, and start over 🙂

Send In The Clones

Once your hosts are installed, it’s time to shut them down and boot them again, this time in headless mode. This is where fixing the boot order on ProxZZ pays off. With all 3 VBVMs are started up, you are ready for the next stage of the exercise: installing your PVE hosts.