Adventures in Proxmox Part 3: Chris don’t know shit about networking

When I first started messing with Proxmox, I crashed my home network.

I have since spent the last several months learning about Proxmox networking using virtual box. I have also been working on a parallel project: upgrading my home network to be segregated using VLANs. Like my budget for server hardware, my budget for network gear is practically nonexistent, so I have been doing a lot of reusing things that should have been replaced years ago.

After a bit of consternation, I settled on a prosumer router and a smart switch, rather than a PC-based router and a managed switch. Mostly because I needed this to work for the family as well as for the lab, and I didn’t want to spend weeks relearning Cisco. Time to tear down the old home network!!

A New Router

My plan is to have 4 “real” networks for my “physical” equipment:

  1. The family’s wireless network – for phones, tablets, game consoles, and tv sticks.
  2. My wired network for my personal workstations and servers.
  3. A VOIP network for POE phones, ATAs, and my PBX.
  4. A server and network lab for me to wreck things.

When I say “real” I really mean “operated by humans” or perhaps “not a Proxmox host”. When I say “physical” I also mean “operated by humans” or perhaps “not a Proxmox host”. At least half of these “real” ports are VLANs, and at least half of these “physical” devices are VMs. In this scenario, “real” and “physical” networks and devices are the ones that I and the family use, compared to the networks that are dedicated to the Proxmox cluster.

The critical distinction is that all of these network segments connect to a different port on the router, and have firewall rules to keep them from connecting to each other. In this scenario, a dumb switch plugged into each port of the router will provide a physically separated network at layer 2 (Ethernet) and a logically separated network at layer 3 (IP). It is here that I have used my first batch of dumb old mini switches:

  1. eth1 – Family Wireless, 192.168.10.0/24
  2. eth2 – Personal Wired, 192.168.11.0/24
  3. eth3 – VOIP, 192.168.12.0/24
  4. eth4 – Lab, 192.168.13.0/24

The family wireless network consists of 2 wireless access points, both with 4 dumb gigabit Ethernet ports:

  1. WAP port 1 -> eth1 on the router, uplink to the Internet
  2. WAP port 2 -> eth0 on the NAS appliance
  3. WAP port 3 -> port 1 on the smart switch
  4. WAP port 4 -> port 1 on the other WAP

So, I had my router set up, and plugging a laptop in to each dumb switch let me pull an IP from the DHCP server for the respective network segment. I was also able to browse the Internet. Awesome. I have managed to convert a big, clunky, error-prone network into four smaller error-prone networks. This is progress?

As far as the family is concerned, eth1 on the router is the network. Wireless access to both the Internet and to the data and media stored on the NAS. If I never plug in the smart switch then only I would notice. I have the WAP’s dumb switch plugged in to the smart switch because I have a media server VM on the Proxmox cluster that I want to put onto the wireless network to stream video to tablets, mobile phones and smart TVs. Because the cluster nodes only have 4 network ports, I need to put multiple network connections on to 1 of those network ports. This is where VLANs come into play. This is also where upgrading my knowledge of routing, switching, and firewalls comes in to play with Proxmox: putting the cluster onto all 4 of my network segments using just one network port from each node.

VLANs: everything you hate about dozens of dumb switches, plus virtualization

With the new router working, it’s time to configure the networks’ core: the smart switch.

VLANs are a great way to divide up a big physical switch into smaller virtual networks. A 24 port switch could be broken down into 4 networks, with a a varying number of ports in each network. You can also put a single switch port onto more than one VLAN. The network traffic gets put into the appropriate virtual network by using tags. You can even put a given port into “all” of the VLANs, this is sometimes referred to as a “trunk.” Trunks are used to connect multiple switches together, passing all tags between them.

Dumb switches can’t tag traffic. So, if you want to mix a smart switch that does VLANs with a dumb switch that doesn’t, you need to make sure that your untagged traffic is going out of the right ports. In the hypothetical 24 port managed switch in the example above, if you put port 2 into VLAN 2, and then plug a dumb switch into port 2, then port 2 needs to know what to do with untagged traffic. Traffic coming out of the dumb switch won’t have tags, and traffic going into to the smart switch will lose its tags. This is the essence of “VID” and “PID/PVID”. A VID is a VLAN ID, PVID is a Port VLAN ID. All the ports on the smart switch need to treat all traffic as tagged, even when it’s not. Untagged traffic needs to be treated differently than tagged traffic, basically meaning that “untagged” is just a special category of “tagged”. The PVID is a kind of “untagged == special tag” way for ports to deal with untagged traffic. This is the exact moment that I developed a migraine.

Star Trek guy with severe head pain.I have done a decent job keeping the family wireless network packets away from everything, and everything away from the family by putting each network segment on its own dumb switch. Now it is time to blur those boundaries a bit by plugging each of those dumb switches into the smart switch. My network is broken into 4 subnets, so my VLANs will break down something like this:

  • VLAN 1 – Family Wireless
  • VLAN 2 – Personal Wired
  • VLAN 3 – VOIP
  • VLAN 4 – Lab

I probably don’t need a separate /24 (class C) network for each VLAN, but I am not very clever and I have zero confidence in my ability to design networks or IP schemes. I know how routing works when you are using /24’s so for my implementation VLAN == /24. Also, as I learned in the Virtual Box lab, network designs get real confusing real fast, so having the VLAN tag roughly correspond to /24 subnet helps me to not go completely insane.

The smart switch is configured by a web interface. This interface has a default IP of 192.168.0.1, so I set a static IP on the Ethernet port of my laptop and logged in. This part of the configuration is important, and it will come into play again later. Once I have all the VLANs set up, I still need to be able to access the switch on this IP address.

I configured the first 4 ports on the switch as access ports or up-links to the dumb switches. Because the dumb switches don’t tag traffic, I needed the uplink ports to treat all “untagged” traffic as tagged to a single VLAN, using the PVID:

  • switch port 1 – VLAN 1, PVID 1
  • switch port 2 – VLAN 2, PVID 2
  • switch port 3 – VLAN 3, PVID 3
  • switch port 4 – VLAN 4, PVID 4

So now, if I change port 5 to VLAN 1 and PVID 1, I can plug in my Windows laptop and pull an IP from the wireless network. Then I can change port 5 to VLAN 2 and PVID 2, and now I can pull an IP from the wired network. Now I need to figure out how to get my Prox cluster nodes to sit on all 4 networks at the same time using a single switch port for each node.

Enter the Management Workstation

Up to this point, I was able to set up my dumb switches and my VLANs with a Windows laptop. I just disabled the WiFi and plugged the Ethernet adapter into the various switches and ports. This was fine for scenarios where one switch port corresponded to just one network segment. But it turns out that Windows can’t do VLANs without proper hardware and software support for the NIC. If you have a VLAN-aware NIC and the Intel or HP enterprise app to configure it, I guess it works fine, but there is no Windows 10 app for the Intel NIC in my crashtop.

In my Virtual Box Proxmox lab, I learned that life is just easier when you have a Linux box dedicated to managing the cluster and testing your network setup, so I decided that before I set up the cluster, I should set up a “Management Workstation.” For the BoxProx lab, I used a Virtual Box VM running a GUI to administer the cluster because I needed a browser on the host only network. Technically, I could have run the management workstation without a GUI and just used SSH tunneling to access the web management interfaces for the Proxmox VMs, but I didn’t want to spend any time doing stupid SSH tricks. I also don’t have the actual hardware cluster running yet, so I need to do this with actual hardware. The hope is that once I get the VLANS and network bridges configured, the workstation will be superfluous. Therefore, the workstation doesn’t have to be powerful at all. Literally any old laptop or desktop that is laying around will do nicely.

My operating system of choice is Turnkey Linux Core. Set up an old desktop on port 5 of the smart switch. For the initial install, I left port 5 configured for VLAN 1 and PVID 1. I was able to pull an IP address from the wireless network, install and update the OS, and configure SSH.

Remote access is important because I can’t sit in my basement all day; Internet access is important because I need to install some network tools.

First step is to get the VLAN tools installed:

apt-get install vlan

Then enable VLAN support in the kernel:

echo 8021q | tee -a /etc/modules

Then add your tagged network interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

Then reboot the machine. I know there is a bunch of crap that you can do to avoid that, but this is the only way I can be sure that it works. I also know that if you name the interface eth0.N you probably don't have to mark the 'vlan-raw-device' but the Debian VLAN tutorial did it so I did it too.

What this does is change the IP of untagged interface eth0 to 192.168.0.10 (remember the IP of the switch from before?) and add eth0.1 (VLAN 1) with an IP of 192.168.1.10 and configured a default gateway and DNS for that interface.

Now, the machine should still be connected to the Internet, and you can modify port 5 on the smart switch to be in VLAN 1 and PVID 1.

If you can ping the IP for the smart switch (192.168.0.1), the IP of something on your wireless network (like an access point) as well as Google's DNS (8.8.8.8) then you are in good shape.

At this point, I left the basement and went upstairs. I connected my laptop to the family wireless network (192.168.1.0/24) to SSH into the workstation. Since I will be making modifications to the smart switch configuration, as well as the management workstation, I decided to configure PuTTy to drop a local port and forward it to 192.168.0.1:80 so that I can access the web interface of the smart switch from my laptop, and the unencrypted HTTP traffic will be secured by the SSH tunnel.

Now I just need to move the Internet access to the 'Lab" VLAN and add the remaining VLANS to /etc/network/interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0

auto eth0.2
iface eth0.2 inet static
    vlan-raw-device eth0
    address 192.168.2.5
    netmask 255.255.255.0

auto eth0.3
iface eth0.3 inet static
    vlan-raw-device eth0
    address 192.168.3.5
    netmask 255.255.255.0

auto eth0.4
iface eth0.4 inet static
    vlan-raw-device eth0
    address 192.168.4.5
    netmask 255.255.255.0
    gateway 192.168.4.1
    dns-nameservers 8.8.8.8 8.8.4.4

The last step is to make sure that smart switch port 5 is part of VLANs 1, 2, 3, and 4, with PVID 1. If all goes well, the workstation can ping the smart switch IP, Google DNS, and servers on all 4 VLANs.

The next step is to use this same network setup for the management NIC on the Proxmox cluster. Using the 4 VLAN interfaces for the network bridges (VMBR1-VMBR4).

The Drama With My New Laptop: the High Cost of Saving $350 (part 2)

This post contains a lot of profanity. Like a shitload.

When we last left our heroes, I had finally gotten Windows working on an SSD after trying a bunch of things, and then basically giving up and then reinstalling everything. Now that the SSD was working, the time had come to encrypt the SSD.

I am a fan of block crypto. I encrypt lots of things, not because I am worried about the government seizing my gear (well, not *that* worried) but because gadgets get lost and stolen. I lost my mobile phone a couple of years ago, and if I hadn’t encrypted it, it would have been nerve wracking worrying about what someone might do with the data that’s on it. So rather than worry about what is or isn’t protected, I just encrypt the whole drive. Full drive encryption is important because Physical Access is Total Access. I have rescued untold amounts of data for others from their crashed or otherwise misbehaving hard drives by removing them and plugging them into a different computer. I don’t normally encrypt the drives on my gaming rigs because if the FBI or whomever needs my Goat Simulator game saves that badly, they are welcome to them. This was a special case because it’s a gaming laptop. My rule is that if it leaves the house, it has to be encrypted.

Modern computers use UEFI to “securely” boot the operating system. I guess this is a security measure to prevent someone from booting your laptop from a CD and stealing all your shit, but since this laptop doesn’t have a Trusted Platform Module, Secure Boot doesn’t protect you from someone plugging your drive into another computer and stealing all your shit, I think it’s more trouble that it’s worth. If you have to ask Windows for permission to boot off a CD, it’s just going to stop the user from doing what he or she wants, it will not stop Proper Villainy(tm).

My favorite disk encryption tool, TrueCrypt, vanished under mysterious circumstances. I won’t get into the conspiracy theories behind its demise, but I have decided to keep encrypting my drive, and that leads me to the next chapter of this saga, where I get punished for using the basic version of Windows.

Part 2 – Solid State Drama’s Revenge

I prefer to run Windows on laptops because of all the bullshit proprietary hardware that goes into them. I am probably showing my age here, but there was a time when hardware support in Linux was spotty. I have swapped out Intel WiFi card for an Atheros cards in laptops to make sure I can do packet injection, but I now have a dedicated Kali laptop for that sort of thing. For my daily driver/EDC laptop, life is just easier with Windows. I know that that fucking with Linux makes a lot of dudes feel superior, and they probably are. For me, I prefer to use Linux for specific tasks (i.e. Kali and Clonezilla) or for servers. With that being said, I am not such a Windows fanboy that I care about the differences between Windows versions. My personal laptop won’t be joining an Active Directory domain, so I just go with whatever version came with my laptop, which I replaced with whatever version MS let me download when I migrated to the SSD.

This path of least resistance philosophy led me to entertain thoughts of using BitLocker to encrypt my hard drive, only I am not running Windows 8.1 Professional or Enterprise, so I guess that BitLocker isn’t included with my version. There is no fucking way that I’m forking over $150 for a new version of Windows after working so hard to save $200 on the RAM and SSD. No TrueCrypt? Fine. No BitLocker? Whatever. I don’t give a fuck. I’ll just use a fork of TrueCrypt called VeraCrypt. Well, VeraCrypt’s boot loader doesn’t play nicely with UEFI and GPT partitions. It only works on MBR disks. feelsbadman.jpg

So after days of messing with various tools to get Windows working on my SSD, and then enduring the hassle of setting up Windows all over again, and waiting on my Steam library to download again, I am faced with yet another hard disk challenge: converting my GPT partitioned drive to MBR without deleting anything. Honestly, now that Steam is in the Debian repos, I am sorely tempted to make my next gaming rig run Linux.

I tried a bunch of things and ended up using the pirated AOMEI tool to do the conversion, and it worked, sort of. The drive booted, and VeraCrypt didn’t bitch about GPT anymore. However, when I went to back up the drive one last time before encrypting it, I discovered that AOMEI half-assed the conversion. According to Clonezilla, my drive had some remnant of the GPT boot stuff left on it that I had to fix with the Linux version of fdisk for GPT, a.k.a gdisk. I have screwed up plenty of working partitions with fdisk, so I was nervous to say the least. Also, the magical -z option that I needed to was buried in the “expert” menu section (AKA Here There Be Dragons!) which added to the danger. Clonezilla said to run gdisk -z but -z isn’t a valid option from the command line.

I read this tutorial to figure out what had to be done, and in the end I just closed my eyes, clenched up my butt cheeks, and hit enter. I got it working, and thankfully I had already made plenty of backups, just in case. Speaking of backups, I should find a way to make running Clonezilla easier…

Update 8/16 – A few months ago, I tried migrating to Win10, but it was a shitshow. I just pirated Win10 Pro (thanks to KMSPico portable, JFGI) and used BitLocker without a TPM. This was less stressful since I set up easy bare metal backups in Part 3.

Stay tuned for the thrilling conclusion in Part 3 – Making Backups Easy to do is Hard 🙂

The Drama With My New Laptop: the High Cost of Saving $350 (part 1)

This post contains a lot of profanity. Like a shitload.

I bought a new laptop a month ago, which for me is like moving to a new apartment. Getting it set up the way that I want it has been a total pain in the ass. Mostly because I have decided to save money by implementing key features myself, but also because the relentless march of progress in the PC market has left me behind. This was an uncharacteristic purchase for me, but I wanted a powerful laptop that I could write, code, play games, and run multiple VMs on. In short, I violated my first rule of personal computing, which is to use dedicated computers for specific tasks.

The goals were:

  1. Be made mostly of aluminum – my laptops tend to have case or hinge problems before they have actual hardware problems, although when they do have hardware problems, it’s almost always the hard drive.
  2. Be ready for anything – have 16gb of RAM, an SSD, USB3.0 and a high end GPU
  3. Have a big screen and full size keyboard – this is replacing a full-sized laptop
  4. Have ample storage – I also bought a caddy to go into the CDROM bay to house a second hard drive
  5. Be encrypted – I normally don’t keep important things on laptops, or gaming rigs, but this is my main computer now
  6. Be backed up regularly – I am not usually a stickler for backups because I use several computers. But with this machine, I want to be able to do a full disk image fairly easily

I have built enterprise servers in less time than I have spent tweaking this fucking laptop. I have more or less achieved all of my goals at the considerable expense of my time and possibly my sanity. There are three major sources of my discontent. The first is that copying a Windows install to a smaller drive is wildly difficult and Asus makes the process even more so. The second, is that Modern versions of Windows are not very friendly with the block crypto tools that I trust. The third is that because I decided to remove the optical drive, I wanted dual-boot Windows with my favorite cloning tool, Clonezilla.

Part 1 – Solid State Drama
I went with the Asus N550jx because it is a mostly aluminum mid-range gaming laptop with a big screen, full size keyboard with keypad, and a touch screen. I can sort of take or leave touchscreens on laptops, but my wife is a fan. I like for she and I to have the same model of laptop. That way, when she runs into problems, I am already very familiar with the hardware and software she is using. The N550jx comes in two models: one with 8GB of RAM and a 1TB mechanical HDD, and one with 16GB of RAM and a 240GB SSD. Both models have the same processor, GPU, screen, and case, and I was able to price another 8GB of ram and a 250GB SSD for almost half the price of the difference between the two models, for a savings of roughly $200. It was a mistake brilliant idea!

#5 Torx bits? On a 6lb laptop? Who does that?Getting the upgrades installed was a series of misadventures. The first obstacle was that for no good goddamn reason, Asus decided to use #5 Torx screws on the chassis. I have plenty of star bit screw drivers from working on Compaq computers back in the Dark Ages, but no #5’s. So what any red-blooded All American Man would do. First, I went on the Internet and complained, and then I ordered yet another set of screwdriver bits from Amazon.

holy shit! i got it working!With the SSD and RAM in place, it was time to get the OS off the mechanical drive onto the SSD. In the past, moving an install of Windows was simply a matter of shrinking partitions with GParted and cloning them with Clonezilla. With the Asus N550jx and Windows 8.1, there is a bunch of bullshit associated with hidden restore partitions with weird flags and whatnot. It is this bullshit that thwarted my countless attempts to migrate the partitions correctly. I even used pirated copies of notable commercial disk cloning tools like Norton Ghost and AOMEI with little success. After a few days of trial and error, I ended up just doing a clean install of Win8 on the SSD. Fortunately, Microsoft lets you create your own install media from an activated Windows system, and Asus is kind enough to make drivers and utilities available on their website for download. So after much installing of software, I had a working OS on the SSD.

All of this trial and error is why I am a huge fan of bare metal backups. I have used all manner of tools and other nonsense to back up Windows and/or data, and the only thing that is truly reliable is dumping the entire drive to an image file on a separate drive. Copying data always leads to missed files, and snapshots and restore points become corrupted especially when malware is involved. Rolling an infected PC back to a restore point is the fastest way to get rid of malware, so most crackers wipe out your restore points as part of the exploit process. Because of this, I don’t really care about recovery partitions, or restore points, or any of that other bullshit. If my laptop eats itself, I just want to roll it back to where it was just before the last time I tried to do something stupid to it. I understand that your typical consumer isn’t familiar with imaging hard drives, and that is why those other tools exist, but for me it’s Clonezilla or GTFO.

Stay tuned for Part 2: Solid State Drama’s Revenge 🙂