Building a Proxmox Test Cluster in VirtualBox Part 4: Containers, Storage, and Replication

In the last installment in this series, I showed you how to build a cluster with separate interfaces dedicated to the cluster heartbeat traffic and to virtual machine migration. In order to see these in action, you need for your PVE hosts to run some VMs of their own. Once there are a three VMs running, we can play with the really great features of a Proxmox Cluster, like storage replication and high availability.

During the initial setup of the Proxmox hosts on VirtualBox, you probably saw an error message about KVM virtualization. I dismissed it as not a big deal. The truth is that this whole “virtualization inside of virtualization” exercise won’t produce any useful PVEVMs. I haven’t done much troubleshooting, but I am fairly certain that while the PVEVMs boot up and you can log into their consoles, they don’t really talk to each other or your physical internal network. That’s not as cool as I had hoped, but the point of this exercise was to figure out the network design for a Proxmox Cluster, not to play with cool double-layer virtual machines. I am sure that you can so some cool router Kung Fu to get them going for real, but for this exercise, PVEVMs booting up is enough. We’re just going to move them around the cluster so see how it all works.

Also, because each PVE cluster node is woefully under-powered, RAM is really at a premium:

So even if they did work as advertised, the PVEVM’s probably wouldn’t work well. Linux Containers are awesomely efficient, especially with RAM, but they can’t squeeze blood from a stone.

Downloading Templates and Building Containers

To build your PVEVMs you need to download a container template. It can be any of them. I prefer the LXC Debian 9 template or the Turnkey Linux Core template. How you want to go about building the test PVEVMs is your business, but the goal is 3 PVEVMs up and running. At different points in this exercise there will be one PVEVM running on each PVE cluster node, and all 3 PVEVMs running on one node. It’s up to you if you want to build them out:

• Download the template to one node, build the container, and then migrate it to another node, OR
• Download the template to each node and build each PVEVM from there, OR
• Download the template, build the container, and then clone it.
• The process is low as hell no matter how you slice it.

Once you have decided your approach, build 3 new containers. It takes a long time, but it demonstrates your need for a central data store for non-VM files. This is where a NAS would be handy. You could set up the NAS to store ISOs, container templates, and backups so that they would be accessible to all the nodes.

It’s important to note that you should build privileged containers. You do this by UN-checking the UN-privileged box. It’s stupid; I know.

At this point, you can migrate a container from one node to the other, but it takes a long time because you have to wait for the container to completely shut down, and then for the container files to completely copy from one cluster node to the next. On real hardware, this process will probably go a bit faster, but this is a good illustration of why we need storage replication.

Storage Replication

Before we can enable replication, we need to set up the ZFS storage properly. In the Building the Cluster Nodes post, we set up a ZFS array called ZStore, and now it’s time to set it up for the whole cluster.

In the Storage View for the cluster you have the option of adding storage. Here you will add a ZFS type and include the zstore/vmdata. Make sure to add all 3 nodes. I called mine “ZVMZ.” At this point it should be apparent that a RAID1 mirror, that is going to be replicated to 2 other mirrors is probably overkill in terms of redundancy, so you should probably do something different on your real hardware. If you have small but fast disks, you might RAID0 them to get nice write speeds, then replicate them for redundancy. Or do whatever.

Once the ZFS storage is set up for all of your nodes, you can move the storage for your PVEVMs to the ZVMZ storage. This will take a long time as well because everything has to copy over. This should be the last time you have to sit through a full copy of anything. Now we can set up replication.

Replication is done on a per VM and per host basis. So you will want to make sure that each VM has a job created to replicate to the other nodes. You only need two jobs for each VM. If you migrate a VM to another host, the replication job will update. The first time the job runs it will take a while. There isn’t a progress bar or anything, so you will have to check the ZVMZ storage on each node to make sure that there are copies of all of your VMs.

With replication set up, even if a cluster node fails, you will only lose 15 minutes (assuming you went with the default schedule) worth of data on the server and you can start up your server on another node with the snapshot. Migrate some VMs and see for yourself, and stay tuned for the next installment: High Availability.