Dan Harmon on the Sexuality of Fallout 4

At my core, I am a tabletop role player. I play a lot of video games, but table top RPGs are my jam. To paraphrase an obscure Fight Club trailer, after tabletop DND, playing video games is “like watching porn when you could be having great sex.”

I so when I *ahem* play solo video games, there is some part of me that wants to put a backstory to either my choices or the choices the game makes for me. When I would marry Mjoll the Lioness in Skyrim and her boy Aerin showed up with her, I figured we were doing some kind of Viking age polyamory type thing.

In Fallout, you get the option to romance a number of your companions, and so I just assumed that my dude was a pan-sexual version of Captain Kirk. Once I loaded the mod that keeps your wife from dying, and she becomes a companion that you can romance… well it was Mjoll the Lioness all over again.

Apparently Dan Harmon also plays FO4 and he agonizes over the choice to romance the companions, and his guilt over it is hilarious.

My Corona Virus Experience

A couple of years months have passed since the initial COVID-19 lock down, and my personal bout with the illness. Now the darkest timeline has moved on to protesting, I thought I would put down my thoughts about my experience before moving on to something more important, like how Black Lives Matter.

Looking back on the events, I don’t think that our healthcare system was prepared for what happened to us. It was impossible for me to get tested for COVID, even with the help of a doctor. I was very fortunate to be able to keep working. Having served in the Ohio Army National Guard, I did some disaster relief work in Nicaragua. There I experienced both tropical disease and life in a failed state. My COVID-19 experience was eerily similar on both fronts.

My Daughter Probably Had COVID-19 A Month Before Anyone Took It Seriously
On February 17th, my 7 year old daughter became extremely ill. She had flu symptoms, including extremely high fevers, but she tested negative for the flu. She was admitted to the hospital on the 24th of February, when her temperature hit 105 degrees Fahrenheit. She remained there for 4 days. She was ultimately treated for pneumonia and given two different rounds of antibiotics. She came home late on the 27th. This was two days before the first confirmed U.S. Corona Virus death in Seattle. The story is that it concentrated in the New York City area, but we live in Cincinnati.

I took most of the week off of work to help keep an eye on my daughter at the hospital. In our family, we have a “no one stays in the hospital alone” policy. My 2 year old son went to school the whole time, but we kept him home the Friday after my daughter came home from the hospital.

The first week of March, my daughter returned to school, and on the 12th of March, my wife and I decided to keep the kids home from school for the rest of the week because of all the school closings. On March 15th, the schools officially shut down. That day, I started feeling ill. The kids have yet to return to school.

I Got Turned Away From The ER With A High Fever
The week of March 15th, my son and I were pretty sick. I was completely exhausted, I had a cough, and I kept getting fevers. Whenever I took Acetaminophen or Ibuprofen, I would get these terrible sweats, followed by intense chills. By Thursday, March 19th, my wife took my son to the ER. Later that night, my temp hit nearly 104 degrees Fahrenheit and I went to the ER as well. I was triaged in a tent in the parking lot, and I sat in a bed for 20 minutes before a doctor told me I probably had it, but they couldn’t test me. Like, get the fuck out basically. That visit cost me $800 after my insurance, BTW.

I was sent home with an information packet about Corona Virus, and told to quarantine for 2 weeks. Then next 15 or so days were awful. I would get these coughing fits where I couldn’t
catch my breath. Everything smelled and tasted awful. My son was on an intense antibiotic that gave him diarrhea, and changing his diapers would make me gag and dry heave. Once my fever cleared up, was still exhausted and I still had a horrible cough. At one point, we had groceries delivered to our front door. I carried them to the kitchen and I got so out of breath that I started vomiting. It felt like times that I had overdone it while sprinting or weight lifting.

My wife, my son, and I all got sick in within two weeks of my daughter’s return home from the hospital. My daughter did not get sick again. It would appear that we had whatever she had, and that she had some sort of immunity.

I began working from home on the 15th of March, and I didn’t go into the office again until mid April, out of sheer necessity. I had to oversee a fiber optic network install for a new office.

I never got a COVID test; my antibody test came back negative

One of my trips into the office, I got a prescription for COVID-19 testing. I saw a testing place near Dayton, but it was always closed down when I went by. I tried calling the Ohio Board of Health about where to go, or how to schedule. They didn’t answer the phone. When they called me back they referred me to the Butler county board of health, who also did not answer. When they called be back, they didn’t know where testing was happening.

Meanwhile, a convicted rapist and a tiger at The Bronx Zoo was able to get a COVID test. This whole situation felt like a massive failure on all levels of the government and healthcare system. And I came through relatively unscathed. I was incredibly fortunate. So many other are not.

Two weeks ago, I got an antibody test that came back negative. My family doctor is thoroughly convinced that my daughter had COVID-19, and that the rest of us probably did too.

It probably won’t go away, and if it does, it will probably come back stronger

Between the botched response to the pandemic and the pressure to reopen the economy, I think we are virtually guaranteed to see either low to no decrease in new cases, and/or a second wave of infections. If the Spanish Flu was any indication, the second wave is probably going to be even worse. Between the 100k+ deaths so far in the US, the economic collapse, and the absolute cash grab by corporations following the bailout, I can’t help but feel like the United States has descended into a failed third-world state like Brazil or Venezuela. Oh, yeah, and during all of this, the two presidential candidates with plans for healthcare reform got pushed out of the primary. FANTASTIC.

Also, those 100k+ deaths are disproportionately affecting poor people and people of color. Of course it does. That has to be the most American thing that I have ever heard.

With the death toll in the U.S. ratcheting up past 100,000 and the estimated mortality rate of somewhere between 1% and 3%, that means that somewhere between ten and thirty million people have gotten the virus. Those numbers are staggering. Somewhere close to the number of seasonal Flu cases each year. Flu is so prevalent that there is a major push each year for people to get the annual vaccine, and sometimes I get the flu anyway. Right now, there is no COVID19 vaccine.

That was my COVID19 experience, which was basically reliving a bout with a disease in a Third World country, only this time it was the US. The only thing missing from this occasion is the leftist guerrillas.

Fallout 4 in the time of COVID19 part 2: Mod Madness

Now that I am at the 2 month mark of quarantine, I have gone more than a little crazy… with Fallout 4 mods.

I loaded up ‘Sim Settlements: Rise of the Commonwealth‘ over the weekend, and it’s pretty cool. It’s a kind of autopilot for building out settlements. Settlements are an important part of the game, because they are a source of money and materials that I need to progress through the game. They are also a quick way to spend 40 levels or so building shotgun shacks for people who complain all the time about not having any beds. Also, I am not super creative with settlements, so I end up building the same things over and over.

ROTC puts the settlers to work building everything themselves. All I have to do is supply them with food, water, liquor, and drugs. The theory is that now I can spend less time building shacks and more time rolling down the streets of the Commonwealth shooting people in the head. ROTC isn’t quite the optimal build experience I was hoping for. This has nothing to do with the quality of the mod, and everything to do with the way I play Fallout.

My two main trading hubs are Sanctuary Hills and The Castle. I basically divide the ‘Wealth into two hemispheres. In the western half, all trade goes to Sanctuary. In the eastern half, traders go to The Castle. I eventually build out all of the settlements with vendors and work benches and hit them up as a traverse the ‘Wealth. It’s kind of reminiscent of The Walking Dead. The two hubs are linked together by a trader (usually Sheffield) and as I pick up new settlements, I send one settler to the closest hub to make building out the settlement that much easier.

Once I have those two trading hubs going, I’m in business, and the other settlements pretty much fall into place. In ROTC you select a settlement leader and the settlers go to work scrapping things and building stuff. The results are these awesome looking post-apocalyptic junk-towns full of crazy little nooks to explore. Overall, it’s pretty awesome.

There are a few problems though; and they lead me to loading more mods.

Problem #1: The Settlers scrap all my shit

There is warning box that literally tells you this is going to happen. I don’t know what I expected.

Once I pull the trigger, they literally knock the whole place down. Including all of the things I built to get the settlement off the ground INCLUDING THE GODDAMN ARTILLERY!! Both my little martial arts and crafts space at Sanctuary and the field artillery at The Castle disappear the second I tell them to get to work.

So if I let the settlers build out all of the settlements, then I have to find a place for me to do my thing. I am sure that if I knew more about ROTC, or city plans, or something, I could solve the problem The Right Way(tm) but that’s really not my style.

In the beginning, I used the Red Rocket Truck Stop as my main trade hub, and devoted the other settlements to being junk-towns. This worked fine until I realized I also needed an eastern trading hub. The solution of course was to use another mod.

The Red Rocket Redone settlements mod turns every Red Rocket into its own small settlement. I was doing this anyway with the Conquest Camping mod to serve as a kind of overflow housing for when my settlements were getting crowded. Now, I am doing the reverse. The Red Rocket mod makes the Red Rockets better suited for settling than Conquest, and I can take them over early on in the game. Now, as I move across The Commonwealth, I gain these buildings as support bases.

With these Red Rockets now under my protection, I can have ready access to workbenches and the like without hunting for them in the crazy junk-town settlements. I can also put artillery at each one to get fire support when I need it. Sure, I have to build out the settlement a bit to support the settlers that I dedicate to trading and gunnery, but if I can keep it small and simple, I can probably do beds in the Red Rocket, and maybe an additional shack for the settlers and put the rest to work trading between settlements. Plus the Red Rockets tend to have all the crafting stations without needing to build them. This is important early in the game because it takes a while to get the perks I need to build workshops myself. Now it’s fairly easy to pop in just about anywhere on the map take a nap, scrap stuff, and craft things.

Problem #2: The Settlers grow the wrong shit

As much as I love not having to plant tons of crops, this does impact the supply of crops that I actually care about. A motivating factor for building settlements is that they produce money and salvage. But they also produce crops. Corps are great for keeping the settlers from bitching about being hungry, but they also have two other distinct uses:

1. You can sell food at vendors for additional caps.
2. You can turn specific crops (purified water, corn, mutfruit, tatos) into vegetable starch, which you can use as adhesive to create just about every weapon or armor mod.

I know it’s probably not very appetizing for the settlers to live on a steady diet of superglue ingredients, but I need scopes and shit for my rifles so I can fight for their freedom goddammit!

So the next solution builds on the first, which is to grow glue components at my Red Rocket settlements with the help of robots via the Mister Gardener mod. Now, when I turn up a Red Rocket, I can outfit it with a couple of food bots to grow my starch components. There is a suite of bot mods available from the author, so I went ahead and loaded them all because I just love robot pets. I especially love the Mr. Law mod, that puts a Protectron on guard to help defend the place.

Problem #3: The Settlements sell the wrong shit

Another benefit of having settlements is being able to sell off loot and stock up on ammo and useful scrap. Vendors will eventually appear in the junk-towns, but it’s only after a lot of upgrades. When I was building out Red Rocket settlements using Conquest, I just put a weapons vendor there so I could sell off loot and buy ammo. Now that the Red Rocket Settlement mod makes them act more like real settlements, I can put more vendors there and collect some caps as well. It’s not a bad way of doing things, since I have Red Rockets set up as trading hubs anywa. It’s like I have a chain of franchises: The Red Rocket Trading Company. These trading posts are getting kind of advanced though, so building defenses is now becoming a priority. I wanted to park one of my companions at each one to help with defenses, but…

Problem #4: The Settlements suck up all my companions

I like to roll with a whole crew when I do my thing: Dogmeat, Warmachine, and a companion. Unfortunately, ROTC requires a companion to serve as city leader to oversee the construction of the junk-town, which basically confines the companion to the settlement. Obviously, that kind of restricts my ability to use companions for either my traveling entourage, or as security for my Red Rockets. So did what I always do, and I loaded a few more mods.

One idea was to add Nora as a companion and travel with her exclusively. She is cool, but because you get a new perk for reaching the topmost level of affinity with a companion, there is an opportunity cost associated with not taking on new companions.

I also thought about trying to load the male version of Nora, the Nate companion, and fully lean into the idea of my survivor being this pansexual polyamorous version of Captain Kirk, just banging everyone in The Commonwealth, but both mods depend on your choice of gender at the beginning of the game. So much for my statement against societal and gender norms.

Then I happened upon Nobody’s Leaders which lets you use named settlers, like Sturges or Ronnie Shaw in place of a companion. This lets me put a named settler in charge of each settlement and I can go back to roaming The Commonwealth and either helping, murdering, or seducing everyone I meet. Then, once I have extracted all the value that I can from them, I dump them at one of my numerous properties around the wasteland to guard farmers and shit. It sounds very predatory when I say it that way.

Sim Settlements and Rise of the Commonwealth have significantly modified my game experience in Fallout 4. Which is very welcome, because I don’t really have the mental or emotional space for a new game right now. My family is currently playing the new Animal Crossing, and I don’t even have room for that.

Fallout 4 in the time of COVID19

It’s been a whole year month of working from home, plus a three month week bout of the virus itself. I have been playing video games, mostly FO4, to cope with the stress.

I am on another play through, this time with mods. I loaded some simple ones, like the Unofficial Fallout 4 patch and the Castle Walls Restored mod, which doesn’t really affect game play that much, other than maybe making the castle easier to defend.

The two game-play affecting mods that I have been running are the Everyone’s Best Friend mod, and the Conquest mod.

Everyone’s Best Friend lets you have a companion and Dogmeat at the same time. As companions go Dogmeat isn’t as good as a humanoid. He can’t use a gun, his melee attacks don’t do as much damage, and he can’t carry as much as a humanoid. Also, even though he can’t be killed, I still feel like shit when he gets hurt. According to the mod, a case could be made that Dogmeat was intended to travel alongside a companion, like Meeko from Skyrim. Without the mod, I just place him at Sanctuary or Red Rocket where I presume he gets taken care of by the settlers.

With the Best Friend mod, a humanoid companion, plus the Sentinel add-on, I have a whole entourage accompanying me around the wasteland. Dogmeat serves as the early warning system; He barks when he locates an enemy. Then, Warmachine rushes in once I start shooting. I still get killed on occasion, but for those little encounters with random Raiders or Ferals, it goes a lot faster. Dogmeat’s scouting is important when I put myself and my companion into suits of power armor and then we roll around the Commonwealth with Warmachine like a small bipedal tank division.

I have blown up the Brotherhood and the Institute enough times that it’s not really about the story anymore, it’s more about the Zen of building up the Minutemen and the settlements. In that vein, the Conquest mod makes for an interesting take. Essentially, the mod equips you with a camping kit so that you can create a little settlement anywhere you want. You can create a little cooking stove, a sleeping bag, tent, and a portable generator for hooking up a construction light. It’s a fun little way to rest up, scrap some equipment, without needing to return to a settlement. I don’t do it all the time, but it’s a good way to stay focused on a quest line, and not get pulled into fixing settlement problems all the time.

The other thing that you can construct at a campsite is a workshop that turns the site into a settlement. You can create up to 10 ‘outposts’ this way. I like to create them a couple of miles from my official settlements to act as a kind of overflow area for the busy/happy settlements that reach capacity fairly quickly. Stores at these outposts let you buy and sell, but they don’t produce caps as well as they do at actual settlements, so I skip the General Stores and Bars, which are cash cows for settlements, and stick to weapons vendors so that I can restock on ammo.

As for locations, I like to use existing structures, like Red Rockets. These tend to have workbenches already in place, so mostly I just need to put in beds, crops, and water. If there are beds set up in these places already, they don’t count for the happiness of the settlers.

You also have to build out defenses because they will get attacked, by both the natural spawns in the area and by the random settlement events. I think it’s fun to put them not far from trouble spots, like College Square or the Quincy Ruins. I then tool up settlers before I move them out to the outposts so I can drop by on occasion to watch the fireworks. Another fun thing to do is build the camps on the military checkpoints after the Institute has been defeated. You have settlers standing by to help the Minutemen during their events, and you have Minutemen to help defend the settlers during their events. Also, it feels good, from a role play standpoint, to build more and better defenses and shelters for the Minutemen at the checkpoint. These dudes are just standing around in the elements 24 hours a day, waiting to get roughed up by god knows what.

I am also interested in the Sim Settlements mod, though I haven’t loaded it up yet. My goal is to find create a gameplay experience that is bascially Animal Crossing with guns.

Adventures in Proxmox Part 3: Chris don’t know shit about networking

When I first started messing with Proxmox, I crashed my home network.

I have since spent the last several months learning about Proxmox networking using virtual box. I have also been working on a parallel project: upgrading my home network to be segregated using VLANs. Like my budget for server hardware, my budget for network gear is practically nonexistent, so I have been doing a lot of reusing things that should have been replaced years ago.

After a bit of consternation, I settled on a prosumer router and a smart switch, rather than a PC-based router and a managed switch. Mostly because I needed this to work for the family as well as for the lab, and I didn’t want to spend weeks relearning Cisco. Time to tear down the old home network!!

A New Router

My plan is to have 4 “real” networks for my “physical” equipment:

  1. The family’s wireless network – for phones, tablets, game consoles, and tv sticks.
  2. My wired network for my personal workstations and servers.
  3. A VOIP network for POE phones, ATAs, and my PBX.
  4. A server and network lab for me to wreck things.

When I say “real” I really mean “operated by humans” or perhaps “not a Proxmox host”. When I say “physical” I also mean “operated by humans” or perhaps “not a Proxmox host”. At least half of these “real” ports are VLANs, and at least half of these “physical” devices are VMs. In this scenario, “real” and “physical” networks and devices are the ones that I and the family use, compared to the networks that are dedicated to the Proxmox cluster.

The critical distinction is that all of these network segments connect to a different port on the router, and have firewall rules to keep them from connecting to each other. In this scenario, a dumb switch plugged into each port of the router will provide a physically separated network at layer 2 (Ethernet) and a logically separated network at layer 3 (IP). It is here that I have used my first batch of dumb old mini switches:

  1. eth1 – Family Wireless, 192.168.10.0/24
  2. eth2 – Personal Wired, 192.168.11.0/24
  3. eth3 – VOIP, 192.168.12.0/24
  4. eth4 – Lab, 192.168.13.0/24

The family wireless network consists of 2 wireless access points, both with 4 dumb gigabit Ethernet ports:

  1. WAP port 1 -> eth1 on the router, uplink to the Internet
  2. WAP port 2 -> eth0 on the NAS appliance
  3. WAP port 3 -> port 1 on the smart switch
  4. WAP port 4 -> port 1 on the other WAP

So, I had my router set up, and plugging a laptop in to each dumb switch let me pull an IP from the DHCP server for the respective network segment. I was also able to browse the Internet. Awesome. I have managed to convert a big, clunky, error-prone network into four smaller error-prone networks. This is progress?

As far as the family is concerned, eth1 on the router is the network. Wireless access to both the Internet and to the data and media stored on the NAS. If I never plug in the smart switch then only I would notice. I have the WAP’s dumb switch plugged in to the smart switch because I have a media server VM on the Proxmox cluster that I want to put onto the wireless network to stream video to tablets, mobile phones and smart TVs. Because the cluster nodes only have 4 network ports, I need to put multiple network connections on to 1 of those network ports. This is where VLANs come into play. This is also where upgrading my knowledge of routing, switching, and firewalls comes in to play with Proxmox: putting the cluster onto all 4 of my network segments using just one network port from each node.

VLANs: everything you hate about dozens of dumb switches, plus virtualization

With the new router working, it’s time to configure the networks’ core: the smart switch.

VLANs are a great way to divide up a big physical switch into smaller virtual networks. A 24 port switch could be broken down into 4 networks, with a a varying number of ports in each network. You can also put a single switch port onto more than one VLAN. The network traffic gets put into the appropriate virtual network by using tags. You can even put a given port into “all” of the VLANs, this is sometimes referred to as a “trunk.” Trunks are used to connect multiple switches together, passing all tags between them.

Dumb switches can’t tag traffic. So, if you want to mix a smart switch that does VLANs with a dumb switch that doesn’t, you need to make sure that your untagged traffic is going out of the right ports. In the hypothetical 24 port managed switch in the example above, if you put port 2 into VLAN 2, and then plug a dumb switch into port 2, then port 2 needs to know what to do with untagged traffic. Traffic coming out of the dumb switch won’t have tags, and traffic going into to the smart switch will lose its tags. This is the essence of “VID” and “PID/PVID”. A VID is a VLAN ID, PVID is a Port VLAN ID. All the ports on the smart switch need to treat all traffic as tagged, even when it’s not. Untagged traffic needs to be treated differently than tagged traffic, basically meaning that “untagged” is just a special category of “tagged”. The PVID is a kind of “untagged == special tag” way for ports to deal with untagged traffic. This is the exact moment that I developed a migraine.

Star Trek guy with severe head pain.I have done a decent job keeping the family wireless network packets away from everything, and everything away from the family by putting each network segment on its own dumb switch. Now it is time to blur those boundaries a bit by plugging each of those dumb switches into the smart switch. My network is broken into 4 subnets, so my VLANs will break down something like this:

  • VLAN 1 – Family Wireless
  • VLAN 2 – Personal Wired
  • VLAN 3 – VOIP
  • VLAN 4 – Lab

I probably don’t need a separate /24 (class C) network for each VLAN, but I am not very clever and I have zero confidence in my ability to design networks or IP schemes. I know how routing works when you are using /24’s so for my implementation VLAN == /24. Also, as I learned in the Virtual Box lab, network designs get real confusing real fast, so having the VLAN tag roughly correspond to /24 subnet helps me to not go completely insane.

The smart switch is configured by a web interface. This interface has a default IP of 192.168.0.1, so I set a static IP on the Ethernet port of my laptop and logged in. This part of the configuration is important, and it will come into play again later. Once I have all the VLANs set up, I still need to be able to access the switch on this IP address.

I configured the first 4 ports on the switch as access ports or up-links to the dumb switches. Because the dumb switches don’t tag traffic, I needed the uplink ports to treat all “untagged” traffic as tagged to a single VLAN, using the PVID:

  • switch port 1 – VLAN 1, PVID 1
  • switch port 2 – VLAN 2, PVID 2
  • switch port 3 – VLAN 3, PVID 3
  • switch port 4 – VLAN 4, PVID 4

So now, if I change port 5 to VLAN 1 and PVID 1, I can plug in my Windows laptop and pull an IP from the wireless network. Then I can change port 5 to VLAN 2 and PVID 2, and now I can pull an IP from the wired network. Now I need to figure out how to get my Prox cluster nodes to sit on all 4 networks at the same time using a single switch port for each node.

Enter the Management Workstation

Up to this point, I was able to set up my dumb switches and my VLANs with a Windows laptop. I just disabled the WiFi and plugged the Ethernet adapter into the various switches and ports. This was fine for scenarios where one switch port corresponded to just one network segment. But it turns out that Windows can’t do VLANs without proper hardware and software support for the NIC. If you have a VLAN-aware NIC and the Intel or HP enterprise app to configure it, I guess it works fine, but there is no Windows 10 app for the Intel NIC in my crashtop.

In my Virtual Box Proxmox lab, I learned that life is just easier when you have a Linux box dedicated to managing the cluster and testing your network setup, so I decided that before I set up the cluster, I should set up a “Management Workstation.” For the BoxProx lab, I used a Virtual Box VM running a GUI to administer the cluster because I needed a browser on the host only network. Technically, I could have run the management workstation without a GUI and just used SSH tunneling to access the web management interfaces for the Proxmox VMs, but I didn’t want to spend any time doing stupid SSH tricks. I also don’t have the actual hardware cluster running yet, so I need to do this with actual hardware. The hope is that once I get the VLANS and network bridges configured, the workstation will be superfluous. Therefore, the workstation doesn’t have to be powerful at all. Literally any old laptop or desktop that is laying around will do nicely.

My operating system of choice is Turnkey Linux Core. Set up an old desktop on port 5 of the smart switch. For the initial install, I left port 5 configured for VLAN 1 and PVID 1. I was able to pull an IP address from the wireless network, install and update the OS, and configure SSH.

Remote access is important because I can’t sit in my basement all day; Internet access is important because I need to install some network tools.

First step is to get the VLAN tools installed:

apt-get install vlan

Then enable VLAN support in the kernel:

echo 8021q | tee -a /etc/modules

Then add your tagged network interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

Then reboot the machine. I know there is a bunch of crap that you can do to avoid that, but this is the only way I can be sure that it works. I also know that if you name the interface eth0.N you probably don't have to mark the 'vlan-raw-device' but the Debian VLAN tutorial did it so I did it too.

What this does is change the IP of untagged interface eth0 to 192.168.0.10 (remember the IP of the switch from before?) and add eth0.1 (VLAN 1) with an IP of 192.168.1.10 and configured a default gateway and DNS for that interface.

Now, the machine should still be connected to the Internet, and you can modify port 5 on the smart switch to be in VLAN 1 and PVID 1.

If you can ping the IP for the smart switch (192.168.0.1), the IP of something on your wireless network (like an access point) as well as Google's DNS (8.8.8.8) then you are in good shape.

At this point, I left the basement and went upstairs. I connected my laptop to the family wireless network (192.168.1.0/24) to SSH into the workstation. Since I will be making modifications to the smart switch configuration, as well as the management workstation, I decided to configure PuTTy to drop a local port and forward it to 192.168.0.1:80 so that I can access the web interface of the smart switch from my laptop, and the unencrypted HTTP traffic will be secured by the SSH tunnel.

Now I just need to move the Internet access to the 'Lab" VLAN and add the remaining VLANS to /etc/network/interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0

auto eth0.2
iface eth0.2 inet static
    vlan-raw-device eth0
    address 192.168.2.5
    netmask 255.255.255.0

auto eth0.3
iface eth0.3 inet static
    vlan-raw-device eth0
    address 192.168.3.5
    netmask 255.255.255.0

auto eth0.4
iface eth0.4 inet static
    vlan-raw-device eth0
    address 192.168.4.5
    netmask 255.255.255.0
    gateway 192.168.4.1
    dns-nameservers 8.8.8.8 8.8.4.4

The last step is to make sure that smart switch port 5 is part of VLANs 1, 2, 3, and 4, with PVID 1. If all goes well, the workstation can ping the smart switch IP, Google DNS, and servers on all 4 VLANs.

The next step is to use this same network setup for the management NIC on the Proxmox cluster. Using the 4 VLAN interfaces for the network bridges (VMBR1-VMBR4).

Building a Proxmox Test Cluster in VirtualBox Part 5: Shit Happened; Lessons Were Learned

Jesus, it’s been almost a year since I posted part 1 of this series.

Hacking stuff is one of the ways that I cope with depression. Like going to the gym and getting stronger, learning new skills is a productive activity that improves my mind and my career. Also like going to the gym, hacking stuff requires a certain level of energy and focus. When I am having a depressive episode, I just can’t make myself do much more than watch TV. I have emerged from my Fallout 4 binge and I am eager to get this hardware cluster off the ground.

Learning Lessons

In my pursuit of a working Virtual Box + Proxmox cluster (Boxmox? ProxBox? BoxProx!) I discovered a few fatal flaws:

  • My testbed is a single laptop, and I used static IP’s that sat on my internal wireless network.
  • That meant that I could hack and demo the cluster at home, but not out in the world, like at Cinci2600.
  • Ergo, the “Management interface sitting on the internal network” question that I excluded from the exercise should not have been excluded.
  • Thus, the laptop-based lab for this project was missing a few things:
    1. 3 “Host Only” networks for the management interface, cluster network, and migration network.
    2. A router VBVM to route traffic bound for the Internet via a NAT interface.
    3. A management workstation VBVM with a GUI, for managing the router and the BoxProx CLI and UI.

The reason that I have been doing all of this in Virtual Box, is because it’s easy to recover from these sorts of mistakes. You can think of this exercise as the “Lab Before The Lab”, or the development phase, before going to an actual hardware lab. I actually gave up on keeping my lab environment separate from my home network because I was always limited by one thing or another. At this point, it’s as much lab as it is production, pretty much everywhere.

Shit Happening
Another component of this exercise that I have not documented is the redesign of my home/lab network to accommodate the new cluster. The old “cluster” is down to two old Proxmox servers that aren’t clustered together. It works for getting shit done for the family (PBX, Plex, Bittorrent, OpenVPN, etc.) but it’s not optimal, nor is the network sufficiently segregated to my satisfaction. So, as I have been doing this, I have also been upgrading the home network and learning more about things like VLANs.

So, the material of the first 4 parts of the series is valid, I just wanted to include the router and workstation bits, which you will probably only need if you want your lab to be portable, and work on wireless networks other than your home.

Modification to the network design

In the first installment, I recommended using a bridged adapter for the management interface. This worked great at home, but once I went anywhere else, the wheels fell off the whole process. I tried things like adding a static IP to my wireless adapter in Windows, and I came to the conclusion that Windows just doesn’t do virtual networking like it’s supposed to.

Hal turns on a light, but the bulb is broken. He takes a new light bulb from the shelf, but the shelf is also broken.

So, when you build your PVE hosts, use 3 host only networks, and use a router VM to connect the cluster to the Internet. Also be sure to disable the DHCP service on all of your host-only networks, like so:

The router

I know I have made simple routers from Debian VMs but for this experiment I spent a fair amount of time in the weeds. So do yourself a favor and just use PFSense. Yes it’s waaaay overkill for what you want to use it for, but it will route packets between two networks with minimal configuration, and that’s really what you want.

Hal gets a screwdriver to fix the shelf, and the drawer is squeaky. He picks up the WD40 but it's empty.

  1. Put the first interface of the PFSense VM on a NAT network.
  2. Make sure to disable the DHCP server on your host-only network interfaces.
  3. Put the second interface for the PFSense VM on the FIRST host-only network interface.
  4. Once you have the VBVM booted up, configure the WAN interface on the NIC that was configured by DHCP, and the LAN interface on the other NIC.
  5. Using the console on the router VBVM, configure the LAN for DHCP. Use a small address pool because there will probably be only one DHCP client ever. Using DHCP is an easy way to make sure that you are looking at the right NIC/virtual network.
  6. I can tell you from experience that if you find yourself twiddling with PFSense settings, you are doing it wrong. Just factory reset the config and move on. This is a BoxProx lab, not a PFSense lab.

The Workstation

Ok, so now you have a small network on host-only adapter 1, and router that connects it to the NAT network on your computer. All these NATs make the cluster network portable, but all but useless for anything else. That’s fine. All you want at this point is for your Linux workstation VBVM to access the Internet despite the fact that its only network interface is sitting on a host-only network.

Lois asks Hal to fix the light bulb and he is under the car yelling.

For the management workstation, you don’t need more than a browser and an SSH client, so literally any distro will work for you. I am a Debian guy, so when I want a no-frills GUI workstation with zero time spent configuring, I use one of the Ubuntu breeds meant for low end computers, like Lubuntu or Ubuntu Mate.

Regardless of the distro, you will be doing some repetitive typing in SSH. On Windows, I recommended MobaXTerm so you can type into multiple terminals at the same time and feel like a super hacker. In the Linux world, the app that you want to use is called “Terminator”. Like everything else on this blog, there is way more to Terminator that I won’t bother with. Just know that you can split your term into two equal parts horizontally and vertically by right clicking, and you can turn on and turn off broadcasting to all your keystrokes by pressing ALT+A and ALT+O respectively. Sorry Terminator/TMux/TWM fans, but I got shit to do.

This phase of the lab is a success if you can boot your Linux VBVM and use a browser to access Google as well as the web UIs for PFSense. You are now free to begin the lab again from Part 1.

Fallout 4 gets me through the holidays

I have been playing Fallout 4 for several months now. The holidays are very stressful for me, so I decided to pick up some creation club add-ons to spice things up.

My last play thru, I sided with The Minutemen against the Institute, took on The Brotherhood of Steel, and freed the traders in Nukaworld. Minutemen vs. The Brotherhood is a spectacular way to finish the main quest line. Also, removing the slave collars from the traders caused everyone’s outfits to glitch, turning Nukaworld Bazaar into a nudist colony. Enjoy your freedom you kinky bastards!

This time thru I sided with The Railroad. The quest line is ok, if a bit stressful. There were moments where I thought I had accidentally backed The Institute.

I also picked up the sentinel power armor addon, the settlement ambush kit, and some of the free armor and weapon skins. I normally don’t go for DLC, but it was a little Christmas present to myself. Skinning armor is nice because it unifies the paint scheme for disparate pieces of armor, which makes your outfit look nice even though the pieces are mismatched. Surprisingly, it’s not that big of a deal for my player character, but it’s nice when outfitting settlers and provisioners. Putting Minutemen or Railroad logos on mismatched armor helps me to not accidentally shoot friendlies during raids. Seeing an armed person walking down the road in Minuteman armor is nice from a role-play standpoint. Like order is being restored to The Commonwealth.

The Sentinel Power Armor

The Sentinel armor is interesting in that it effectively adds a second companion. Much like how Skyrim let you have both a dog and a human companion, this lets your “pet” be a full suit of power armor. You can equip it like a companion, and it’s default weapon is full auto laser rifle. Probably the best feature is the personality mode, which lets you choose between the Protectron, Assaultron, or my personal favorite: Mister Gutsy. My nickname for him is “War Machine.”

Having a companion with you long term can get annoying when you hear them say the same things over and over, especially Preston’s judgmental ass criticizing me for picking up scrap. You know that sniper scope on that bad ass rifle you carrying? I made it with junk I be scavenging. I put this shit to good use, Pres, so shut up. The Mister Gutsy personality option doesn’t talk much, it just mostly accuses all hostiles of being Communists. It’s pretty awesome, except when I am trying to creep up on a target to back stab it and he suddenly yells “IS THAT SOMEONE THAT NEEDS ME TO KICK THEIR ASS!?!?!”

You can load the sentinel up with gear too, but once it’s been outfitted with a full set of power armor, his carry capacity isn’t great, and it’s a pain accessing the menu for it, so I just use it in extreme emergencies. Also, the sentry armor doesn’t get damaged the way that wearing power armor can, so there is way less to maintain. Putting a companion in power armor sounds good in theory, but they get shot up and you end up repairing their shit all the time. I am not sure if the quality of the armor matters for the sentinel, so I just throw basic armor with Minuteman or Railroad paint on War Machine and roll out.

My current play thru is on “Very Hard” mode, which means that a lot of enemies could one-shot me at the lower levels. Having a War Machine with me is nice because he rushes in to the fight and draws out enemies so I can snipe them from a covered position. He repairs himself, but not very quickly, so it’s possible for him to get shutdown, and you have to physically access his console to jump start the repair process. This is different than having a robot companion that you can use a repair kit on. It’s not as fast as using a repair kit, but it doesn’t take any materials. Once you get used to his cover fire, you can notice real quick when he and the companion are down, because suddenly everything is shooting at you.

Automatron

Speaking of robot companions. I think that my favorite expansion is Automatron. I am a sucker for pet robots, and building a robot to protect a settlement is one of my favorite things to do in the game. I especially love the Mister Handy torso, and using it to make my own varieties of Mister Handy, like one with giant caliper hands and pincer legs that I call “Mister Pinchy”, or one with giant saw blades for hands and buzz saw legs that I call “Mister Slicey”. Other models include “Mister Shooty” who has minigun arms, and “Mister Tradey” who has all of the storage mods, and works as a Provisioner.

I also built a Sentry Bot for The Castle that I named “Sarge” after the malfunctioning robot in The Castle basement. I wish you could repair him specifically so that he could have a real personality. But he looks pretty cool rolling around The Castle keeping his big red eye on things. The other thing that would be great is to put faction paint on robots. Mister Shooty would look pretty awesome with a Minuteman paint job.

Once you have done the Mechanist quest line, you end up with Jezebel as a kind of settler. She refuses to interact positively with you, so while you can assign her to jobs at a settlement, and she will do them, she makes a lousy shop keeper because you can’t buy or sell anything. She just complains about you. I just put her to work at Graygarden as either a farmer or security. I would send more robots to live and work at Graygarden, but I think that having a bunch of companions at one settlement is a waste. I like to send Codsworth to live there, until Ada shows up, then I send him off to another settlement. I guess I could return him to Sanctuary and send Preston to The Castle, but I kind of like having Preston at Sanctuary for some reason.

I don’t know if having multiple companions at a settlement affects how many settlers that you can attract, but I like to wait until a settlement is maxed out population-wise before I add robots. Companions seem to make better security personnel than regular settlers, so that is the job I usually give them. Good security becomes a major deal when you start provoking raids with the Settlement Ambush Kit.

Settlement Ambush Kit

The settlement ambush kit adds a couple of cool things. You can add walls and a special guard tower to your settlement, which makes defending it a lot simpler because your guards stay in one place, rather than roaming around. You can also add remote view video cameras that let you kind of fast travel from one site to another without actually leaving. A funny glitch with the sentinel armor is that it will physically go to the site you are viewing, so switching camera views makes War Machine run all over the place to stand in front of your camera.

A really cool feature of the kit is the ability to send out fake distress signals that trigger raids. So far I have only fired it up once, but it just sends wave after wave of raiders to your settlement, and it keeps score of how many waves you have survived. I am assuming there is one for ferals as well, but I haven’t tried it yet. I re-rolled my character not long after getting it and right now my settlements are ill-equipped for a massive raid.

The Tipping Point

Now that I have done two full re-rolls, I can say with relative confidence that the game balance shifts when your character level hits the mid 40’s. If you have been doing settlement building and Minutemen quests consistently, they should be producing caps, food, and salvage to the point that you are crafting most of the things you need (oil, adhesive, stimpaks) and mostly buying ammo, aluminum, and steel. My first play-though went over 100th level, and the game was fairly easy to play at that point. I had settlements that had nuclear reactors, multiple industrial water purifiers, and laser turrets protecting everything. The main story line can put you in front of Kellogg pretty quickly, and he can be really tough to beat at low levels. Having multiple combat perks combined with high end weapons and armor make a big difference.

I think a challenging play through would be to use no companions or Sentinels, and to skip the Minutemen all together. You will still end up with settlements, but you probably wouldn’t have nearly as many. I might try that when I re-roll again, supporting either the Institute or the Brotherhood of Steel. I think that both of those factions also want some form of settlement, so you may end up with them anyway.

My Life with Multitops: using multiple types of laptops

It’s the end of the year, and I have a lot on my mind. So rather than deal with it, I am going to write about laptops. I have owned many laptops over the years, most of them have been refurbished or re-purposed from some other role. In many ways, I am a bit like a crazy cat lady, but instead of cats, I am surrounded by laptops. I tend to own and operate a few laptops because I have a few specific use cases with different hardware requirements. Rather than calling them laptops, I like to refer to them by the purpose that they serve for me.

  1. TypetopA big laptop that is suited for long typing sessions. In the past I wrote (and hacked, and coded) a lot more than I do now. I used to write papers for school, reports or emails for work, blog posts, or creative works. While my ideal writing environment is an office chair, large monitor and a buckling spring keyboard, any table with laptop that has a full-sized keyboard will do. I don’t consider these large and rather heavy machines to be mobile so much as portable. Of my fleet of laptops, the ones optimized for typing also tend to be the most expensive. This is the model that I normally go for when an employer is picking up the tab.
  2. NotetopA tiny laptop that is suited for note taking. I have spent many hours in lecture halls and the like taking notes for classes. I don’t really use a laptop for notes at work, unless I am the designated minutes-taker, for example when I worked at a startup company out west, or in my time on the board of directors at Hive13. For class room notes, nothing beats a small netbook, especially if you are also carrying around textbooks and paper notebooks. I found that the accessory pocket in a backpack kept the laptop from being smashed by textbooks. It’s too bad that the iPad pretty much destroyed the market for cheap netbooks, because I dearly loved those old MSI’s.
  3. JettopA burner laptop for travel. I used to travel to hacker conferences like DefCon, and you would occasionally need a laptop, but there was always a chance that something awful might happen to it. It might get stolen, it might get confiscated by law enforcement at an international border, it might get hacked by someone with way better skills than mine, or someone [like me] might drunkenly vomit on it or throw it out of a window. To minimize this risk, I would take a cheap laptop with minimal personal information and strong encryption. Once I started carrying a smartphone, I would also travel with an old flip phone, just to be safe. Later on, I would just take my work phone and turn off WiFi and Bluetooth. In later years, I bought a refurbished Chromebook and traveled with it. I found that a Chromebook along with a small Android tablet combined to make a good, lightweight, toolkit.
  4. ShoptopA laptop for hardware hacking. In the years I spent with Hive13, I was always in need of multiple ports to connect to things around the shop. I would use multiple serial or USB ports to connect to hacker hardware like Arduinos or old copiers and printers. Even today I occasionally need to plug in multiple large external hard drives to share pirated goods at events like 2600. In the past, I have found older laptops to be indispensable in these “workshop” environments due to their legacy ports. For me, workshops are also fairly dangerous places, where laptops get exposed to power tool mishaps, fire, and on more than one occasion, blood. It is these dangers, combined with a need for old ports, that I prefer to keep older laptops around, however under-powered they may become. I am not sure what I will do in the future, when even my eldest laptop has only a couple of USB ports. I suppose that a shoptop is the kind of thing that I should probably build myself. I keep wanting to get back into electronics, maybe a DIY shoptop would be a good way to get started.
  5. CrashtopA laptop for network configuration and troubleshooting Pretty much always the secondary function of a shoptop, looking into network crashes pretty much always requires a laptop. For a dude that tinkers with computers, I like to think that I have a decent grasp of networking. Not just cabling, but also routing, switching and even telephones. My home network is as much a lab as it is anything else. My main router has a console port, and while most of the network configuring I do is with SSH or a browser, sometimes you just need a laptop that you can physically plug in to a device. Of all the legacy ports to disappear from a modern laptop, I will miss the gigabit Ethernet port the most. Sure there are USB serial and Ethernet adapters, but those just aren’t the same as having the gear built right in. Also like the shoptop, I often think about either building a device, or maybe refurbishing a vintage device to troubleshoot networks with. I have always wanted a very industrial-looking 80’s device like the old Informer 213 for terminal-type stuff. At one point in my life, I had an old laptop that had a voice modem in it so that I could also mess with analog telephone lines.
  6. I am not in the market for a new laptop just yet. My typetop plays Skyrim and Fallout 4 decently. Plus it’s time for me to get into consoles again 🙂

Being Addicted to Fallout 4

In the past, I have written about playing video games to cope with depression. It’s that time of year again, so I am playing games a lot. I basically love 4 kinds of games:

  1. Open world RPGs with various factions, families, and morality systems (like Skyrim or Fable)
  2. First-Person Shooters with engaging single player stories (like Half-Life)
  3. Farm management games with community, friendship, and/or romance dynamics (like Animal Crossing or Stardew Valley)
  4. Tower Defense games where you manage funds/materials/etc to build steadily stronger fortresses
  5. Pretty much anything where you have robot minions

Fallout 4 is basically a turducken of these various game elements. It’s pretty much the most addictive thing I have ever encountered. Imagine a dish made by the guys from Epic Meal Time, using only ingredients provided by the guys from Breaking Bad. Fallout 4 is basically things I like about Skyrim, dialed up to 11.

I picked up the full Fallout 4 suite on a ridiculous Steam sale a few months ago, and I have spent pretty much all of my non-sleeping, non-working, and non-child-rearing hours playing it. I know the game is like 5 years old. My gaming hardware is also 5 years old. Fight me.

In Skyrim, I loved helping kids and dogs. I basically forgot about the dragons and focused on amassing gold in order to build a house for everyone. Then it turned out that Lucia is afraid of the swamp where I built the house, so I had to win a civil war for her so we can live in peace and safety in Windhelm.

Well, in Fallout 4, not only are you searching for your lost son, you meet Dogmeat within the first 10 minutes and he’s way more bad ass than Meeko. I did a ton of work to ensure that Dogmeat was safe at Sanctuary Hills, under the watchful eyes of Codsworth, while I searched for Shaun.

Another thing I loved about Skyrim was meeting, marrying, and traveling with Mjoll the Lioness. She was a total bad ass, and so she and her dude Aerin come to live with me, the kids, the dog, and the House Carl in some kind of weird Nordic polyamory version of The Brady Bunch.

Well, in Fallout 4, I was able to seduce Preston. We took on the raiders, ferals, and supermutants of the commonwealth while building settlements together. *Then* I was also able to seduce Piper, Curie, and Hancock. I was like some kind of post-nuke/pan-sexual version of Captain Kirk, getting in fights with and/or boning robots and shit. Again, taking something I liked from Skyrim and turning the volume up to 11.

As much as the memes liked to dunk on Preston for never shutting up about helping settlements, settlements fucking rule. Which is the other way that Fallout 4 got me: Building. Fucking. Farms. I set up a bunch of settlements, planted crops to feed the settlers, and built shacks and shit for them to live in. Oh, and I surrounded them with automated turrets. There’s nothing greater than hearing on the radio that a settlement needs help, just to fast travel there and watch the attackers get shredded by my sentries.

Life in The Commonwealth is much easier when you have large supplies of ammunition and caps. A holdover from my Skyrim days is my tendency to sneak about, shooting targets from a distance. My survivor is a decent sniper, but he really only gets to clip a couple of targets at proper sniper range. After that, he has to creep up a bit closer. At sniper range, I like to use a .50 cal hunting rifle. While closing the gap I like to use a custom .308 combat rifle that I call “Quickshot.” It’s great for putting things down with two or three rounds, and it’s silenced. If I have time to line up a headshot, like in VATS, I can put most things down with one round. The problem is that .50 and .308 ammo is kind of rare so I am constantly purchasing it. One way to make lots of caps and to buy ammo at a discount is to set up vendors at the various settlements. I put up a weapons stand and I can usually buy 50-100 rounds of .308 every couple of days. Every time I come to a settlement to drop off salvage, I hit up the emporium for .308, .50 cal, and shotgun shells.

I tend to hoard .45 and 5.56mm to hand out to settlers that work security. I build out combat rifles and assault rifles for the provisioners and settlers assigned to guard posts and to scavenging stations. These dudes get my hand-me-down weapons and armor, as well as stuff I’ve looted off of Raiders. When a settlement gets attacked all of the settlers will run to fight, so it doesn’t hurt to outfit even the farmers with armor and upgraded weapons. When you are 80th level or so, your settlements can have like 20 people in them, so that’s a lot of gear to be handing out. There are like 20 settlements in the Commonwealth proper, plus the ones for Far Harbor and Nukaworld, which means that the endgame for me is all about dealing arms to your peasant militia.

Another luxury item to have is large amounts of salvage. Most vendors will let you buy large shipments of salvage for a thousand or more caps. When you are maxing out the defenses of a settlement, or building lots of robots, you tend to run low on aluminum, steel, and oil. One way I get steady access to lots salvage is to set up trading emporium at my settlements as well. This lets me buy salvage in bulk. One particular item that you need tons of is adhesive. You can craft vegetable starch at a cooking station by combining corn, mutfruit, and tatoes. So putting folks to work farming these items at your settlements is important. Once you have a large supply of vegetable starch, you can sell off the excess that appears in your workshops. Once your local traders are out of caps, you can go to the Diamond City Market to unload the rest. If you set up a clinic, you can also buy bloodpacks which you will need to make stimpaks. Depending on your selection of perks, you can keep your survivor going on just stimpaks.

The other advantage of numerous settlements is provisioners. With the Local Leader perk, you can add a settler to a trade route. This lets you share the salvage that you have with all of your trading settlements. Provisioners will then walk the roads between their trading settlements on a regular basis. Obviously, this is very handy for building out new settlement, or for getting supplies to smaller settlements, but there are two other advantages:

  1. If you find a provisioner out in the wild, you can dump any excess items or salvage on them, and the items will eventually find their way to a settlement workshop.
  2. If you arm and armor a provisioner, he or she will engage the random spawns that happen out on the road, making The Commonwealth a tiny bit safer for you and your other settlements. I tend to use Sanctuary Hills and The Castle as my main trading hubs. It’s funny when a random attack happens on one of these places when there are half a dozen traders standing around. It’s like having extra security. Building custom arms and armor for provisioners and settlement security is a good way to safely earn XP as well. If you combine building settlement stuff with crafting while abusing the “well rested” perk, you can level yourself a bit without getting killed constantly.

The morality system is fairly strict as well. I have rolled back a game more than once because I chose poorly at a critical juncture. I will go back through and play the other way, as a bad guy or whatever, at some point. I must have played through Skyrim a dozen times trying to create the perfect play thru, or at least as perfect as I can get it before something bugs out 🙂

I am on my third re-roll, each time siding with the Minutemen, working with the Railroad, and against the Institute and the Brotherhood of Steel. I also decided to wipe out the gangs of Nukaworld. Nukaworld is great fun, even if you are being a good guy. Although taking on the gangs does feel a bit genocidal at times.

Building a Proxmox Test Cluster in VirtualBox Part 4: Containers, Storage, and Replication

In the last installment in this series, I showed you how to build a cluster with separate interfaces dedicated to the cluster heartbeat traffic and to virtual machine migration. In order to see these in action, you need for your PVE hosts to run some VMs of their own. Once there are a three VMs running, we can play with the really great features of a Proxmox Cluster, like storage replication and high availability.

During the initial setup of the Proxmox hosts on VirtualBox, you probably saw an error message about KVM virtualization. I dismissed it as not a big deal. The truth is that this whole “virtualization inside of virtualization” exercise won’t produce any useful PVEVMs. I haven’t done much troubleshooting, but I am fairly certain that while the PVEVMs boot up and you can log into their consoles, they don’t really talk to each other or your physical internal network. That’s not as cool as I had hoped, but the point of this exercise was to figure out the network design for a Proxmox Cluster, not to play with cool double-layer virtual machines. I am sure that you can so some cool router Kung Fu to get them going for real, but for this exercise, PVEVMs booting up is enough. We’re just going to move them around the cluster so see how it all works.

Also, because each PVE cluster node is woefully under-powered, RAM is really at a premium:

So even if they did work as advertised, the PVEVM’s probably wouldn’t work well. Linux Containers are awesomely efficient, especially with RAM, but they can’t squeeze blood from a stone.

Downloading Templates and Building Containers

To build your PVEVMs you need to download a container template. It can be any of them. I prefer the LXC Debian 9 template or the Turnkey Linux Core template. How you want to go about building the test PVEVMs is your business, but the goal is 3 PVEVMs up and running. At different points in this exercise there will be one PVEVM running on each PVE cluster node, and all 3 PVEVMs running on one node. It’s up to you if you want to build them out:

  • Download the template to one node, build the container, and then migrate it to another node, OR
  • Download the template to each node and build each PVEVM from there, OR
  • Download the template, build the container, and then clone it.
  • The process is low as hell no matter how you slice it.

Once you have decided your approach, build 3 new containers. It takes a long time, but it demonstrates your need for a central data store for non-VM files. This is where a NAS would be handy. You could set up the NAS to store ISOs, container templates, and backups so that they would be accessible to all the nodes.

It’s important to note that you should build privileged containers. You do this by UN-checking the UN-privileged box. It’s stupid; I know.

At this point, you can migrate a container from one node to the other, but it takes a long time because you have to wait for the container to completely shut down, and then for the container files to completely copy from one cluster node to the next. On real hardware, this process will probably go a bit faster, but this is a good illustration of why we need storage replication.

Storage Replication

Before we can enable replication, we need to set up the ZFS storage properly. In the Building the Cluster Nodes post, we set up a ZFS array called ZStore, and now it’s time to set it up for the whole cluster.

In the Storage View for the cluster you have the option of adding storage. Here you will add a ZFS type and include the zstore/vmdata. Make sure to add all 3 nodes. I called mine “ZVMZ.” At this point it should be apparent that a RAID1 mirror, that is going to be replicated to 2 other mirrors is probably overkill in terms of redundancy, so you should probably do something different on your real hardware. If you have small but fast disks, you might RAID0 them to get nice write speeds, then replicate them for redundancy. Or do whatever.

Once the ZFS storage is set up for all of your nodes, you can move the storage for your PVEVMs to the ZVMZ storage. This will take a long time as well because everything has to copy over. This should be the last time you have to sit through a full copy of anything. Now we can set up replication.

Replication is done on a per VM and per host basis. So you will want to make sure that each VM has a job created to replicate to the other nodes. You only need two jobs for each VM. If you migrate a VM to another host, the replication job will update. The first time the job runs it will take a while. There isn’t a progress bar or anything, so you will have to check the ZVMZ storage on each node to make sure that there are copies of all of your VMs.

With replication set up, even if a cluster node fails, you will only lose 15 minutes (assuming you went with the default schedule) worth of data on the server and you can start up your server on another node with the snapshot. Migrate some VMs and see for yourself, and stay tuned for the next installment: High Availability.