When I first started messing with Proxmox, I crashed my home network. If you aren’t interested in the story of my journey of network sexual awakening, click here.
I have since spent the last several months learning about Proxmox networking using virtual box. I have also been working on a parallel project: upgrading my home network to be segregated using VLANs. Like my budget for server hardware, my budget for network gear is practically nonexistent, so I have been doing a lot of reusing things that should have been replaced years ago.
After a bit of consternation, I settled on a prosumer router and a smart switch, rather than a PC-based router and a managed switch. Mostly because I needed this to work for the family as well as for the lab, and I didn’t want to spend weeks relearning Cisco. Time to tear down the old home network!!
A New Router
My plan is to have 4 “real” networks for my “physical” equipment:
- The family’s wireless network – for phones, tablets, game consoles, and tv sticks.
- My wired network for my personal workstations and servers.
- A VOIP network for POE phones, ATAs, and my PBX.
- A server and network lab for me to wreck things.
When I say “real” I really mean “operated by humans” or perhaps “not a Proxmox host”. When I say “physical” I also mean “operated by humans” or perhaps “not a Proxmox host”. At least half of these “real” ports are VLANs, and at least half of these “physical” devices are VMs. In this scenario, “real” and “physical” networks and devices are the ones that I and the family use, compared to the networks that are dedicated to the Proxmox cluster.
The critical distinction is that all of these network segments connect to a different port on the router, and have firewall rules to keep them from connecting to each other. In this scenario, a dumb switch plugged into each port of the router will provide a physically separated network at layer 2 (Ethernet) and a logically separated network at layer 3 (IP). It is here that I have used my first batch of dumb old mini switches:
- eth1 – Family Wireless, 192.168.10.0/24
- eth2 – Personal Wired, 192.168.11.0/24
- eth3 – VOIP, 192.168.12.0/24
- eth4 – Lab, 192.168.13.0/24
The family wireless network consists of 2 wireless access points, both with 4 dumb gigabit Ethernet ports:
- WAP port 1 -> eth1 on the router, uplink to the Internet
- WAP port 2 -> eth0 on the NAS appliance
- WAP port 3 -> port 1 on the smart switch
- WAP port 4 -> port 1 on the other WAP
So, I had my router set up, and plugging a laptop in to each dumb switch let me pull an IP from the DHCP server for the respective network segment. I was also able to browse the Internet. Awesome. I have managed to convert a big, clunky, error-prone network into four smaller error-prone networks. This is progress?
As far as the family is concerned, eth1 on the router is the network. Wireless access to both the Internet and to the data and media stored on the NAS. If I never plug in the smart switch then only I would notice. I have the WAP’s dumb switch plugged in to the smart switch because I have a media server VM on the Proxmox cluster that I want to put onto the wireless network to stream video to tablets, mobile phones and smart TVs. Because the cluster nodes only have 4 network ports, I need to put multiple network connections on to 1 of those network ports. This is where VLANs come into play. This is also where upgrading my knowledge of routing, switching, and firewalls comes in to play with Proxmox: putting the cluster onto all 4 of my network segments using just one network port from each node.
VLANs: everything you hate about dozens of dumb switches, plus virtualization
With the new router working, it’s time to configure the networks’ core: the smart switch.
VLANs are a great way to divide up a big physical switch into smaller virtual networks. A 24 port switch could be broken down into 4 networks, with a a varying number of ports in each network. You can also put a single switch port onto more than one VLAN. The network traffic gets put into the appropriate virtual network by using tags. You can even put a given port into “all” of the VLANs, this is sometimes referred to as a “trunk.” Trunks are used to connect multiple switches together, passing all tags between them.
Dumb switches can’t tag traffic. So, if you want to mix a smart switch that does VLANs with a dumb switch that doesn’t, you need to make sure that your untagged traffic is going out of the right ports. In the hypothetical 24 port managed switch in the example above, if you put port 2 into VLAN 2, and then plug a dumb switch into port 2, then port 2 needs to know what to do with untagged traffic. Traffic coming out of the dumb switch won’t have tags, and traffic going into to the smart switch will lose its tags. This is the essence of “VID” and “PID/PVID”. A VID is a VLAN ID, PVID is a Port VLAN ID. All the ports on the smart switch need to treat all traffic as tagged, even when it’s not. Untagged traffic needs to be treated differently than tagged traffic, basically meaning that “untagged” is just a special category of “tagged”. The PVID is a kind of “untagged == special tag” way for ports to deal with untagged traffic. This is the exact moment that I developed a migraine.
I have done a decent job keeping the family wireless network packets away from everything, and everything away from the family by putting each network segment on its own dumb switch. Now it is time to blur those boundaries a bit by plugging each of those dumb switches into the smart switch. My network is broken into 4 subnets, so my VLANs will break down something like this:
- VLAN 1 – Family Wireless
- VLAN 2 – Personal Wired
- VLAN 3 – VOIP
- VLAN 4 – Lab
I probably don’t need a separate /24 (class C) network for each VLAN, but I am not very clever and I have zero confidence in my ability to design networks or IP schemes. I know how routing works when you are using /24’s so for my implementation VLAN == /24. Also, as I learned in the Virtual Box lab, network designs get real confusing real fast, so having the VLAN tag roughly correspond to /24 subnet helps me to not go completely insane.
The smart switch is configured by a web interface. This interface has a default IP of 192.168.0.1, so I set a static IP on the Ethernet port of my laptop and logged in. This part of the configuration is important, and it will come into play again later. Once I have all the VLANs set up, I still need to be able to access the switch on this IP address.
I configured the first 4 ports on the switch as access ports or up-links to the dumb switches. Because the dumb switches don’t tag traffic, I needed the uplink ports to treat all “untagged” traffic as tagged to a single VLAN, using the PVID:
- switch port 1 – VLAN 1, PVID 1
- switch port 2 – VLAN 2, PVID 2
- switch port 3 – VLAN 3, PVID 3
- switch port 4 – VLAN 4, PVID 4
So now, if I change port 5 to VLAN 1 and PVID 1, I can plug in my Windows laptop and pull an IP from the wireless network. Then I can change port 5 to VLAN 2 and PVID 2, and now I can pull an IP from the wired network. Now I need to figure out how to get my Prox cluster nodes to sit on all 4 networks at the same time using a single switch port for each node.
Enter the Management Workstation
Up to this point, I was able to set up my dumb switches and my VLANs with a Windows laptop. I just disabled the WiFi and plugged the Ethernet adapter into the various switches and ports. This was fine for scenarios where one switch port corresponded to just one network segment. But it turns out that Windows can’t do VLANs without proper hardware and software support for the NIC. If you have a VLAN-aware NIC and the Intel or HP enterprise app to configure it, I guess it works fine, but there is no Windows 10 app for the Intel NIC in my crashtop.
In my Virtual Box Proxmox lab, I learned that life is just easier when you have a Linux box dedicated to managing the cluster and testing your network setup, so I decided that before I set up the cluster, I should set up a “Management Workstation.” For the BoxProx lab, I used a Virtual Box VM running a GUI to administer the cluster because I needed a browser on the host only network. Technically, I could have run the management workstation without a GUI and just used SSH tunneling to access the web management interfaces for the Proxmox VMs, but I didn’t want to spend any time doing stupid SSH tricks. I also don’t have the actual hardware cluster running yet, so I need to do this with actual hardware. The hope is that once I get the VLANS and network bridges configured, the workstation will be superfluous. Therefore, the workstation doesn’t have to be powerful at all. Literally any old laptop or desktop that is laying around will do nicely.
My operating system of choice is Turnkey Linux Core. Set up an old desktop on port 5 of the smart switch. For the initial install, I left port 5 configured for VLAN 1 and PVID 1. I was able to pull an IP address from the wireless network, install and update the OS, and configure SSH.
Remote access is important because I can’t sit in my basement all day; Internet access is important because I need to install some network tools.
First step is to get the VLAN tools installed:
apt-get install vlan
Then enable VLAN support in the kernel:
echo 8021q | tee -a /etc/modules
Then add your tagged network interfaces:
nano /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.0.10
netmask 255.255.255.0
auto eth0.1
iface eth0.1 inet static
vlan-raw-device eth0
address 192.168.1.10
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 8.8.8.8 8.8.4.4
Then reboot the machine. I know there is a bunch of crap that you can do to avoid that, but this is the only way I can be sure that it works. I also know that if you name the interface eth0.N you probably don't have to mark the 'vlan-raw-device' but the Debian VLAN tutorial did it so I did it too.
What this does is change the IP of untagged interface eth0 to 192.168.0.10 (remember the IP of the switch from before?) and add eth0.1 (VLAN 1) with an IP of 192.168.1.10 and configured a default gateway and DNS for that interface.
Now, the machine should still be connected to the Internet, and you can modify port 5 on the smart switch to be in VLAN 1 and PVID 1.
If you can ping the IP for the smart switch (192.168.0.1), the IP of something on your wireless network (like an access point) as well as Google's DNS (8.8.8.8) then you are in good shape.
At this point, I left the basement and went upstairs. I connected my laptop to the family wireless network (192.168.1.0/24) to SSH into the workstation. Since I will be making modifications to the smart switch configuration, as well as the management workstation, I decided to configure PuTTy to drop a local port and forward it to 192.168.0.1:80 so that I can access the web interface of the smart switch from my laptop, and the unencrypted HTTP traffic will be secured by the SSH tunnel.
Now I just need to move the Internet access to the 'Lab" VLAN and add the remaining VLANS to /etc/network/interfaces:
nano /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.0.10
netmask 255.255.255.0
auto eth0.1
iface eth0.1 inet static
vlan-raw-device eth0
address 192.168.1.10
netmask 255.255.255.0
auto eth0.2
iface eth0.2 inet static
vlan-raw-device eth0
address 192.168.2.5
netmask 255.255.255.0
auto eth0.3
iface eth0.3 inet static
vlan-raw-device eth0
address 192.168.3.5
netmask 255.255.255.0
auto eth0.4
iface eth0.4 inet static
vlan-raw-device eth0
address 192.168.4.5
netmask 255.255.255.0
gateway 192.168.4.1
dns-nameservers 8.8.8.8 8.8.4.4
The last step is to make sure that smart switch port 5 is part of VLANs 1, 2, 3, and 4, with PVID 1. If all goes well, the workstation can ping the smart switch IP, Google DNS, and servers on all 4 VLANs.
The next step is to use this same network setup for the management NIC on the Proxmox cluster. Using the 4 VLAN interfaces for the network bridges (VMBR1-VMBR4).
This post contains a lot of profanity. Like a shitload.
This path of least resistance philosophy led me to entertain thoughts of using BitLocker to encrypt my hard drive, only I am not running Windows 8.1 Professional or Enterprise, so I guess that BitLocker isn’t included with my version. There is no fucking way that I’m forking over $150 for a new version of Windows after working so hard to save $200 on the RAM and SSD. No TrueCrypt? Fine. No BitLocker? Whatever. I don’t give a fuck. I’ll just use a fork of TrueCrypt called VeraCrypt. Well, VeraCrypt’s boot loader doesn’t play nicely with UEFI and GPT partitions. It only works on MBR disks. feelsbadman.jpg
I tried a bunch of things and ended up using the pirated AOMEI tool to do the conversion, and it worked, sort of. The drive booted, and VeraCrypt didn’t bitch about GPT anymore. However, when I went to back up the drive one last time before encrypting it, I discovered that AOMEI half-assed the conversion. According to Clonezilla, my drive had some remnant of the GPT boot stuff left on it that I had to fix with the Linux version of fdisk for GPT, a.k.a gdisk. I have screwed up plenty of working partitions with fdisk, so I was nervous to say the least. Also, the magical -z option that I needed to was buried in the “expert” menu section (AKA Here There Be Dragons!) which added to the danger. Clonezilla said to run gdisk -z but -z isn’t a valid option from the command line.
With the SSD and RAM in place, it was time to get the OS off the mechanical drive onto the SSD. In the past, moving an install of Windows was simply a matter of shrinking partitions with 
chrizzle23.com 
You must be logged in to post a comment.