Remote Access Shenanigans in 2021

A couple of years ago, I wrote about how I get access to my home network. In a previous job, I worked nights for a big financial company with a very restrictive network. I often connect to the work network from home (which I call telecommuting) and to the home network from work (which I call reverse telecommuting). Most of the time it’s to fix stuff, sometimes it’s because there is a downtime window: for work that is at night when everyone has gone home, at home it’s during the day when everyone is at work/school.

My dream is to be able to sit at a desk, anywhere in the world, and do whatever it is that I need to do, with minimal fuss on my part, and with no impact on the people (coworkers and family) that I support. It’s a lofty goal that is beset by overprotective firewalls, pandemics, and crappy laptops.

When in doubt, SSH

Most of my remote administration tasks involve logging in to either a system administration web GUI, or logging into a command shell. For that, SSH tunneling works great. I have port 22 opened on my firewall and mapped to a Linux server. That host does nothing except serve as a jumpbox into my lab network. Once I can SSH in, I can drop a local port to SSH to my management workstation that sits on the other VLANs. The reason I don’t forward port 22 directly to the management workstation is that I have concerns about my internal VLANs being a single hop from the Internet. It’s not really a security measure so much as an obscurity measure.

I haven’t done much traveling in the last 2 years, and on the one trip that I did take, I didn’t have much time for hacker shit. But when I am away from home, and able to do hacker shit, NeoRouter comes in handy.

NeoRouter on a hosted server

I have also written about cloud hosted VMs. Some of these services are fairly inexpensive, but not at all reliable, and some of them are quite reliable, but they are very expensive. I would put Cloud At Cost in the first category, and Digital Ocean in the second. Cloud hosting is an important upgrade to my remote access arsenal, because in a world of NAT and firewalls, having something directly connected to the Internet with a static IP is a game changer.

In my network travels, I came across the free tier of Google Compute Engine. It does what it says on the tin: a shared CPU Linux container with a static IP. It won’t cost you much for the first year, but it is extremely under powered. Fortunately, NeoRouter will provide access to plenty of resources hosted on my Proxmox cluster at home, and the service itself doesn’t take much compute power. After the free year, the VM costs me $4 give or take, sometimes it goes up to almost $6 to run the box 24×7. You can shave off a dollar or so each month by scheduling downtime. For me that was 12:30am to 6:30am. It took me a couple of hours to get it working, which I guess is more about principal than actual savings. If you value your time, just get a Digital Ocean droplet for like $5 and change per month and get on with your life.

With NeoRouter running on a hosted VM, it creates an overlay network that allows my Windows desktops and Linux servers to communicate with each other, even though they are on different physical and logical networks.

I have also begun experimenting with graphical Linux desktops instead of Linux servers or Windows desktops, but I will save that for a later post.

VirtualProx experiments 2021

I’ve written a ton of posts about running a Proxmox cluster in VirtualBox.

Part of why I write these things is to help me record work that I did in the past, kind of like a journal. Part of it is the hope that someone will read it and benefit from it. Mostly, building home lab shit and writing about it is how I cope with… *gestures vaguely*

The new Proxmox 7.X version is out, and the Proxmox Backup server has also been released. So I set up another Proxmox cluster in Virtual Box. Here are some observations and things that I learned from the exercise.

  1. I learned enough about host only networks in Virtualbox to eliminate the need for a management workstation.
    I am a big fan of setting up a workstation with a GUI to test network configurations during the network construction phase. In the old days of hardware that meant using an old garbage PC, even though it was a waste of electricity. Now that we have virtualization, I still put a low powered VM on different subnets for troubleshooting. In those same old days when I was growing my Unix skills, I almost always used a Windows PC with multiple network cards because Windows has historically been completely stupid about VLANs and the like.

    Also, using a workstation with an OS that you are very comfortable with lets you focus on what you are learning. Trying to figure out a new OS while also figuring out networking, or virtualization, or scripting/programming is overwhelming. So, in previous labs, I recommended spinning up a basic VM that sat on the host only network for doing firewall/network administration tasks. Well, no more!

    It turns out that the IP address that you assign in the VirtualBox host network manager app is just the static IP address that your physical host has on that network interface. It’s not any sort of network configuration. I know that should have been obvious, but like the management workstation mentioned above, I am figuring this out as I go.

    So, when you are setting up your host only network interfaces, just pick any IP in the range that you want to use. I love the number 23, so that is the last octet that I pick for my physical host. If you set that IP to something other than .1 or .254 or any IP in your DHCP range, you can use the browser on your host computer to configure the ProxMox cluster. You will still need static IPs and multiple network interfaces for management, clustering, and the like.

  2. Doing hardcore system administration tasks via the web UI has gotten a lot better
    My Unix/Linux skills are decent. Not as great as professional sysadmins, but better than most professional IT types. The same goes for my knowledge of networking and virtualization. I can hold a conversation with the folks that specialize in it. So, when I am trying to figure out ProxMox shit, I prefer the web UI so I am not getting out into the weeds chasing down Linux sytax issues or finding obscure things in config files, which I like to call Config File Fuckery(tm).

    You can configure the IPs for your interfaces with the UI, which didn’t work well in the past. You can also define your VLANs in the web UI, and change their names. I like for the VLAN ID/tag to correspond to the third octet of the assigned IP, so VLAN200 would have an IP of 192.168.200.0/24. Yes you can do vlan0 -> 192.168.0.0/24, but that’s no fun 🙂

    I have not yet figured out how to create a ZFS pool on a host using the web UI. You can create the pool as storage in the web UI, configuring your disks for use in the pool still requires the command line, as far as I can tell.

    Creating the cluster in the web UI is super simple now, but specifying a network for VM migration to another cluster node still requires editing the datacenter.cfg file as outlined in Part 3: Building the Cluster.

  3. Proxmox backup server is just for backups

    Having a dedicated server for hosting backups is a great idea. Normally, I set up an NFS server as shared storage between the nodes, where I put container templates, ISO files, and snapshots of machines.

    Proxmox Backup Server integrates into your Proxmox datacenter as storage, and you can use it as a destination for backups. That part is pretty slick, but you can ONLY set it up as a target for backups.

    The other shared storage stuff, doesn’t look like it’s an option. At least not in the web UI.

    I am sure there is a reason for having one server for backups and another for shared storage, which probably has to do with tape drives. For my use case, I would like to download ISOs and container templates to one place and have it be available to all the cluster nodes, which requires an NFS server somewhere. I also want to use shared storage for backups, which could be a Proxmox Backup server OR the same NFS server that I would need for shared storage.

  4. Running a backup server and a NAS seems like a waste
    I have seen forum posts about mounting an NFS share and using it as the datastore. I was more interested in doing the opposite, which is exporting an NFS share to the cluster nodes. It’s Debian Linux under the hood, and I can absolutely just create a directory on the root filesystem and export it. That’s not the point.

    I have also seen forum posts where users run the backup server as a VM. This is probably the use case for the NFS data store: keeping the files on a NAS and the backup software on a VM. I am contemplating doing the opposite, which is running the backup server on bare metal, and running the file server as a VM. I already have a hardware NAS that I am currently using as the shared storage for my hardware Proxmox cluster.

    In hardware news, I have acquired 3 rackmount servers for my hardware cluster. I don’t have a rack or anything to put them in, so stay tuned for some DIY rack making!

Getting in to Ham Radio

I decided to pick up a new quarantine hobby besides playing Fallout 4: Amateur radio. I won’t be posting a bunch of radio stuff here because I want to dedicate a site specifically to that. I also have done a bit of traveling, and I want to document my adventures there too. I did a similar thing with writing about telephones. I didn’t want to clutter up my personal blog with a bunch of radio or telephone shit.

It turns out that Amateur radio has a lot of overlap with computers. Hams are big into Raspberry Pi’s and I even found a VOIP service specifically for hams!

There is a lot of electronics in amateur radio as well, which is a real weak point for me. I have messed with Arudinos a bit, but I’ve mostly stuck to computers. It’s a fascinating hobby, full of different things to learn about antennas, batteries, and even connecting computers over a radio.

So there was an attempted coup

I thought this 2020 bullshit was over with the arrival of 2021.

I probably said this before, but I don’t see how the next couple of years don’t bring about more sectarian violence. My anxiety since the election began is that we are headed for one of 3 outcomes:

  1. A Trump Win – which would ramp up Black Lives Matter protests even more than this summer, with even more crackdowns.
  2. A Biden Win – which ramps up the 2nd amendment and anti-lockdown protests at state houses and the like <-you are here
  3. No Clear Winner – which results in a full blown civil war.

I was not expecting to be right, or for the bullshit to kick in so fast. It really made my head swim.

Now I have to pack up my shit and be ready to go fight somewhere. I’ve been learning about amateur radio, and I think that I could revive some of my military communications skills and serve as an Radio Telephone Operator for BLM protests and the like. I also had some emergency medicine training, so maybe I could put that to use. I’d really like to get through this without a gun if the universe would allow it.

Machines appearing and disappearing in NeoRouter

I just spent the last half hour scratching my head at a weird problem that I was having with NeoRouter. Two windows hosts kept appearing and disappearing in my NeoRouter network. Both machines could log in successfully, but neither machine could see the other in the list of computers. They seemed to be knocking each other out of the network, as if they were knocking each other off.

It turns out that if you clone a Windows machine with Neo Router pre-installed, you end up with IP conflicts, even if you set different static IPs for each host. So if you decide to clone hosts, be sure that you install Neo Router *after* you clone the hosts.

The Back Story

With my new upgraded VLAN home network, plus my quarantined/working from home/life circumstances, I used to have a desktop computer that was on all the time to support all of my remote access shenanigans. In the old flat network days I had one desktop computer that ran 24×7 and sat on the same network as all of my servers. Mostly the goal of remote access is either:

  1. a shell on a server or router
  2. a webpage on an appliance like a router, switch, or file server or
  3. a desktop on a Windows machine that would then provide me 1 or 2

With my new network design I have two VLANs for my servers:

  1. a DMZ for things that ultimately face the Internet, and
  2. A personal internal network that is visible to neither the family wireless network nor the Internet

If you will recall, I have a network management workstation that I can use as a jump box to get into each segment. However, this host isn’t accessible via the Internet. For that I have a couple of Internet facing hosts that I call ‘hubs’. One host is a bottom tier Google Compute instance, the other is a host sitting in the DMZ with a bunk port forwarded to it. Under the most extreme circumstances, I can tunnel through the Google hub, into the DMZ hub, to get a shell on the network management workstation, where I can either set up a socks proxy for internally hosted web management pages, or drop a remote port for RDP to a Windows host.

NeoRouter

OR, I could just use Neo Router. When the networking gods are smiling on me, my Windows laptop and Windows desktop can talk to each other directly via the NR overlay network. With Neo Router, I can have hosts on different VLANs which are not accessible via the Internet, become accessible to other members of the NR network. When I use Windows or Linux machines that can run browsers, there is no need for Stupid SSH Tricks(tm).

The idea was simple: spin up 2 virtual machines (VMs) running Graphical Desktops (GUIs), one GUIVM on the DMZ network, and one GUIVM on the internal wired network. This way I can do arbitrary tasks sitting on either network by connecting to the appropriate GUIVM. I will call these machines “Portals”. Portal-DMZ will sit on the DMZ network, and Portal-Int will sit on the private internal network.

Since I am spinning these VMs up on Proxmox, I could just build one GUIVM, configure it, and then clone it. I used Windows to get it done fast, but ultimately I would like to conserve RAM by using low powered Linux machines.

Turns out the cloning was the source of my strange problem. Apparently there is some sort of signature that makes each node unique that cannot be duplicated without all hell breaking loose.

Dan Harmon on the Sexuality of Fallout 4

At my core, I am a tabletop role player. I play a lot of video games, but table top RPGs are my jam. To paraphrase an obscure Fight Club trailer, after tabletop DND, playing video games is “like watching porn when you could be having great sex.”

I so when I *ahem* play solo video games, there is some part of me that wants to put a backstory to either my choices or the choices the game makes for me. When I would marry Mjoll the Lioness in Skyrim and her boy Aerin showed up with her, I figured we were doing some kind of Viking age polyamory type thing.

In Fallout, you get the option to romance a number of your companions, and so I just assumed that my dude was a pan-sexual version of Captain Kirk. Once I loaded the mod that keeps your wife from dying, and she becomes a companion that you can romance… well it was Mjoll the Lioness all over again.

Apparently Dan Harmon also plays FO4 and he agonizes over the choice to romance the companions, and his guilt over it is hilarious.

When I play FO4, my survivor identifies as a white male. I also run the “Nora Spouse Companion” mod that allows me to also run Nora. Once I have reached affinity with her and then with Codsworth, I run with Preston. I also have the Gunners vs. Minutemen pack from Creation Club and I take Preston with me for the entire quest, once we have retaken The Castle. 

Between retaking the Castle, helping out settlements, and paying the gunners back for the Quincy massacre, I usually hit full affinity with Preston. I then go for the gusto with the romance options and then finally, I set him up as the leader of his own settlement via Sim Settlements, usually Starlight Drive-in.

Preston has had a rough go of things. He watched his idols, the Minutemen, fall apart. He had all of the soldiers and most of the civilians he was responsible for die on his watch, and he ended up in a siege at the Museum Of Freedom at Concord. It not a stretch to say that when you meet Preston Garvey, it is on the worst day of his life. If anyone deserves love and fire support, it’s Preston.

My Corona Virus Experience

A couple of years months have passed since the initial COVID-19 lock down, and my personal bout with the illness. Now the darkest timeline has moved on to protesting, I thought I would put down my thoughts about my experience before moving on to something more important, like how Black Lives Matter.

Looking back on the events, I don’t think that our healthcare system was prepared for what happened to us. It was impossible for me to get tested for COVID, even with the help of a doctor. I was very fortunate to be able to keep working. Having served in the Ohio Army National Guard, I did some disaster relief work in Nicaragua. There I experienced both tropical disease and life in a failed state. My COVID-19 experience was eerily similar on both fronts.

My Daughter Probably Had COVID-19 A Month Before Anyone Took It Seriously
On February 17th, my 7 year old daughter became extremely ill. She had flu symptoms, including extremely high fevers, but she tested negative for the flu. She was admitted to the hospital on the 24th of February, when her temperature hit 105 degrees Fahrenheit. She remained there for 4 days. She was ultimately treated for pneumonia and given two different rounds of antibiotics. She came home late on the 27th. This was two days before the first confirmed U.S. Corona Virus death in Seattle. The story is that it concentrated in the New York City area, but we live in Cincinnati.

I took most of the week off of work to help keep an eye on my daughter at the hospital. In our family, we have a “no one stays in the hospital alone” policy. My 2 year old son went to school the whole time, but we kept him home the Friday after my daughter came home from the hospital.

The first week of March, my daughter returned to school, and on the 12th of March, my wife and I decided to keep the kids home from school for the rest of the week because of all the school closings. On March 15th, the schools officially shut down. That day, I started feeling ill. The kids have yet to return to school.

I Got Turned Away From The ER With A High Fever
The week of March 15th, my son and I were pretty sick. I was completely exhausted, I had a cough, and I kept getting fevers. Whenever I took Acetaminophen or Ibuprofen, I would get these terrible sweats, followed by intense chills. By Thursday, March 19th, my wife took my son to the ER. Later that night, my temp hit nearly 104 degrees Fahrenheit and I went to the ER as well. I was triaged in a tent in the parking lot, and I sat in a bed for 20 minutes before a doctor told me I probably had it, but they couldn’t test me. Like, get the fuck out basically. That visit cost me $800 after my insurance, BTW.

I was sent home with an information packet about Corona Virus, and told to quarantine for 2 weeks. Then next 15 or so days were awful. I would get these coughing fits where I couldn’t
catch my breath. Everything smelled and tasted awful. My son was on an intense antibiotic that gave him diarrhea, and changing his diapers would make me gag and dry heave. Once my fever cleared up, was still exhausted and I still had a horrible cough. At one point, we had groceries delivered to our front door. I carried them to the kitchen and I got so out of breath that I started vomiting. It felt like times that I had overdone it while sprinting or weight lifting.

My wife, my son, and I all got sick in within two weeks of my daughter’s return home from the hospital. My daughter did not get sick again. It would appear that we had whatever she had, and that she had some sort of immunity.

I began working from home on the 15th of March, and I didn’t go into the office again until mid April, out of sheer necessity. I had to oversee a fiber optic network install for a new office.

I never got a COVID test; my antibody test came back negative

One of my trips into the office, I got a prescription for COVID-19 testing. I saw a testing place near Dayton, but it was always closed down when I went by. I tried calling the Ohio Board of Health about where to go, or how to schedule. They didn’t answer the phone. When they called me back they referred me to the Butler county board of health, who also did not answer. When they called be back, they didn’t know where testing was happening.

Meanwhile, a convicted rapist and a tiger at The Bronx Zoo was able to get a COVID test. This whole situation felt like a massive failure on all levels of the government and healthcare system. And I came through relatively unscathed. I was incredibly fortunate. So many other are not.

Two weeks ago, I got an antibody test that came back negative. My family doctor is thoroughly convinced that my daughter had COVID-19, and that the rest of us probably did too.

It probably won’t go away, and if it does, it will probably come back stronger

Between the botched response to the pandemic and the pressure to reopen the economy, I think we are virtually guaranteed to see either low to no decrease in new cases, and/or a second wave of infections. If the Spanish Flu was any indication, the second wave is probably going to be even worse. Between the 100k+ deaths so far in the US, the economic collapse, and the absolute cash grab by corporations following the bailout, I can’t help but feel like the United States has descended into a failed third-world state like Brazil or Venezuela. Oh, yeah, and during all of this, the two presidential candidates with plans for healthcare reform got pushed out of the primary. FANTASTIC.

Also, those 100k+ deaths are disproportionately affecting poor people and people of color. Of course it does. That has to be the most American thing that I have ever heard.

With the death toll in the U.S. ratcheting up past 100,000 and the estimated mortality rate of somewhere between 1% and 3%, that means that somewhere between ten and thirty million people have gotten the virus. Those numbers are staggering. Somewhere close to the number of seasonal Flu cases each year. Flu is so prevalent that there is a major push each year for people to get the annual vaccine, and sometimes I get the flu anyway. Right now, there is no COVID19 vaccine.

That was my COVID19 experience, which was basically reliving a bout with a disease in a Third World country, only this time it was the US. The only thing missing from this occasion is the leftist guerrillas.

Fallout 4 in the time of COVID19 part 2: Mod Madness

Now that I am at the 2 month mark of quarantine, I have gone more than a little crazy… with Fallout 4 mods.

I loaded up ‘Sim Settlements: Rise of the Commonwealth‘ over the weekend, and it’s pretty cool. It’s a kind of autopilot for building out settlements. Settlements are an important part of the game, because they are a source of money and materials that I need to progress through the game. They are also a quick way to spend 40 levels or so building shotgun shacks for people who complain all the time about not having any beds. Also, I am not super creative with settlements, so I end up building the same things over and over.

ROTC puts the settlers to work building everything themselves. All I have to do is supply them with food, water, liquor, and drugs. The theory is that now I can spend less time building shacks and more time rolling down the streets of the Commonwealth shooting people in the head. ROTC isn’t quite the optimal build experience I was hoping for. This has nothing to do with the quality of the mod, and everything to do with the way I play Fallout.

My two main trading hubs are Sanctuary Hills and The Castle. I basically divide the ‘Wealth into two hemispheres. In the western half, all trade goes to Sanctuary. In the eastern half, traders go to The Castle. I eventually build out all of the settlements with vendors and work benches and hit them up as a traverse the ‘Wealth. It’s kind of reminiscent of The Walking Dead. The two hubs are linked together by a trader (usually Sheffield) and as I pick up new settlements, I send one settler to the closest hub to make building out the settlement that much easier.

Once I have those two trading hubs going, I’m in business, and the other settlements pretty much fall into place. In ROTC you select a settlement leader and the settlers go to work scrapping things and building stuff. The results are these awesome looking post-apocalyptic junk-towns full of crazy little nooks to explore. Overall, it’s pretty awesome.

There are a few problems though; and they lead me to loading more mods.

Problem #1: The Settlers scrap all my shit

There is warning box that literally tells you this is going to happen. I don’t know what I expected.

Once I pull the trigger, they literally knock the whole place down. Including all of the things I built to get the settlement off the ground INCLUDING THE GODDAMN ARTILLERY!! Both my little martial arts and crafts space at Sanctuary and the field artillery at The Castle disappear the second I tell them to get to work.

So if I let the settlers build out all of the settlements, then I have to find a place for me to do my thing. I am sure that if I knew more about ROTC, or city plans, or something, I could solve the problem The Right Way(tm) but that’s really not my style.

In the beginning, I used the Red Rocket Truck Stop as my main trade hub, and devoted the other settlements to being junk-towns. This worked fine until I realized I also needed an eastern trading hub. The solution of course was to use another mod.

The Red Rocket Redone settlements mod turns every Red Rocket into its own small settlement. I was doing this anyway with the Conquest Camping mod to serve as a kind of overflow housing for when my settlements were getting crowded. Now, I am doing the reverse. The Red Rocket mod makes the Red Rockets better suited for settling than Conquest, and I can take them over early on in the game. Now, as I move across The Commonwealth, I gain these buildings as support bases.

With these Red Rockets now under my protection, I can have ready access to workbenches and the like without hunting for them in the crazy junk-town settlements. I can also put artillery at each one to get fire support when I need it. Sure, I have to build out the settlement a bit to support the settlers that I dedicate to trading and gunnery, but if I can keep it small and simple, I can probably do beds in the Red Rocket, and maybe an additional shack for the settlers and put the rest to work trading between settlements. Plus the Red Rockets tend to have all the crafting stations without needing to build them. This is important early in the game because it takes a while to get the perks I need to build workshops myself. Now it’s fairly easy to pop in just about anywhere on the map take a nap, scrap stuff, and craft things.

Problem #2: The Settlers grow the wrong shit

As much as I love not having to plant tons of crops, this does impact the supply of crops that I actually care about. A motivating factor for building settlements is that they produce money and salvage. But they also produce crops. Corps are great for keeping the settlers from bitching about being hungry, but they also have two other distinct uses:

1. You can sell food at vendors for additional caps.
2. You can turn specific crops (purified water, corn, mutfruit, tatos) into vegetable starch, which you can use as adhesive to create just about every weapon or armor mod.

I know it’s probably not very appetizing for the settlers to live on a steady diet of superglue ingredients, but I need scopes and shit for my rifles so I can fight for their freedom goddammit!

So the next solution builds on the first, which is to grow glue components at my Red Rocket settlements with the help of robots via the Mister Gardener mod. Now, when I turn up a Red Rocket, I can outfit it with a couple of food bots to grow my starch components. There is a suite of bot mods available from the author, so I went ahead and loaded them all because I just love robot pets. I especially love the Mr. Law mod, that puts a Protectron on guard to help defend the place.

Problem #3: The Settlements sell the wrong shit

Another benefit of having settlements is being able to sell off loot and stock up on ammo and useful scrap. Vendors will eventually appear in the junk-towns, but it’s only after a lot of upgrades. When I was building out Red Rocket settlements using Conquest, I just put a weapons vendor there so I could sell off loot and buy ammo. Now that the Red Rocket Settlement mod makes them act more like real settlements, I can put more vendors there and collect some caps as well. It’s not a bad way of doing things, since I have Red Rockets set up as trading hubs anywa. It’s like I have a chain of franchises: The Red Rocket Trading Company. These trading posts are getting kind of advanced though, so building defenses is now becoming a priority. I wanted to park one of my companions at each one to help with defenses, but…

Problem #4: The Settlements suck up all my companions

I like to roll with a whole crew when I do my thing: Dogmeat, Warmachine, and a companion. Unfortunately, ROTC requires a companion to serve as city leader to oversee the construction of the junk-town, which basically confines the companion to the settlement. Obviously, that kind of restricts my ability to use companions for either my traveling entourage, or as security for my Red Rockets. So did what I always do, and I loaded a few more mods.

One idea was to add Nora as a companion and travel with her exclusively. She is cool, but because you get a new perk for reaching the topmost level of affinity with a companion, there is an opportunity cost associated with not taking on new companions.

I also thought about trying to load the male version of Nora, the Nate companion, and fully lean into the idea of my survivor being this pansexual polyamorous version of Captain Kirk, just banging everyone in The Commonwealth, but both mods depend on your choice of gender at the beginning of the game. So much for my statement against societal and gender norms.

Then I happened upon Nobody’s Leaders which lets you use named settlers, like Sturges or Ronnie Shaw in place of a companion. This lets me put a named settler in charge of each settlement and I can go back to roaming The Commonwealth and either helping, murdering, or seducing everyone I meet. Then, once I have extracted all the value that I can from them, I dump them at one of my numerous properties around the wasteland to guard farmers and shit. It sounds very predatory when I say it that way.

Sim Settlements and Rise of the Commonwealth have significantly modified my game experience in Fallout 4. Which is very welcome, because I don’t really have the mental or emotional space for a new game right now. My family is currently playing the new Animal Crossing, and I don’t even have room for that.

Fallout 4 in the time of COVID19

It’s been a whole year month of working from home, plus a three month week bout of the virus itself. I have been playing video games, mostly FO4, to cope with the stress.

I am on another play through, this time with mods. I loaded some simple ones, like the Unofficial Fallout 4 patch and the Castle Walls Restored mod, which doesn’t really affect game play that much, other than maybe making the castle easier to defend.

The two game-play affecting mods that I have been running are the Everyone’s Best Friend mod, and the Conquest mod.

Everyone’s Best Friend lets you have a companion and Dogmeat at the same time. As companions go Dogmeat isn’t as good as a humanoid. He can’t use a gun, his melee attacks don’t do as much damage, and he can’t carry as much as a humanoid. Also, even though he can’t be killed, I still feel like shit when he gets hurt. According to the mod, a case could be made that Dogmeat was intended to travel alongside a companion, like Meeko from Skyrim. Without the mod, I just place him at Sanctuary or Red Rocket where I presume he gets taken care of by the settlers.

With the Best Friend mod, a humanoid companion, plus the Sentinel add-on, I have a whole entourage accompanying me around the wasteland. Dogmeat serves as the early warning system; He barks when he locates an enemy. Then, Warmachine rushes in once I start shooting. I still get killed on occasion, but for those little encounters with random Raiders or Ferals, it goes a lot faster. Dogmeat’s scouting is important when I put myself and my companion into suits of power armor and then we roll around the Commonwealth with Warmachine like a small bipedal tank division.

I have blown up the Brotherhood and the Institute enough times that it’s not really about the story anymore, it’s more about the Zen of building up the Minutemen and the settlements. In that vein, the Conquest mod makes for an interesting take. Essentially, the mod equips you with a camping kit so that you can create a little settlement anywhere you want. You can create a little cooking stove, a sleeping bag, tent, and a portable generator for hooking up a construction light. It’s a fun little way to rest up, scrap some equipment, without needing to return to a settlement. I don’t do it all the time, but it’s a good way to stay focused on a quest line, and not get pulled into fixing settlement problems all the time.

The other thing that you can construct at a campsite is a workshop that turns the site into a settlement. You can create up to 10 ‘outposts’ this way. I like to create them a couple of miles from my official settlements to act as a kind of overflow area for the busy/happy settlements that reach capacity fairly quickly. Stores at these outposts let you buy and sell, but they don’t produce caps as well as they do at actual settlements, so I skip the General Stores and Bars, which are cash cows for settlements, and stick to weapons vendors so that I can restock on ammo.

As for locations, I like to use existing structures, like Red Rockets. These tend to have workbenches already in place, so mostly I just need to put in beds, crops, and water. If there are beds set up in these places already, they don’t count for the happiness of the settlers.

You also have to build out defenses because they will get attacked, by both the natural spawns in the area and by the random settlement events. I think it’s fun to put them not far from trouble spots, like College Square or the Quincy Ruins. I then tool up settlers before I move them out to the outposts so I can drop by on occasion to watch the fireworks. Another fun thing to do is build the camps on the military checkpoints after the Institute has been defeated. You have settlers standing by to help the Minutemen during their events, and you have Minutemen to help defend the settlers during their events. Also, it feels good, from a role play standpoint, to build more and better defenses and shelters for the Minutemen at the checkpoint. These dudes are just standing around in the elements 24 hours a day, waiting to get roughed up by god knows what.

I am also interested in the Sim Settlements mod, though I haven’t loaded it up yet. My goal is to find create a gameplay experience that is bascially Animal Crossing with guns.

Adventures in Proxmox Part 3: Chris don’t know shit about networking

When I first started messing with Proxmox, I crashed my home network. If you aren’t interested in the story of my journey of network sexual awakening, click here.

I have since spent the last several months learning about Proxmox networking using virtual box. I have also been working on a parallel project: upgrading my home network to be segregated using VLANs. Like my budget for server hardware, my budget for network gear is practically nonexistent, so I have been doing a lot of reusing things that should have been replaced years ago.

After a bit of consternation, I settled on a prosumer router and a smart switch, rather than a PC-based router and a managed switch. Mostly because I needed this to work for the family as well as for the lab, and I didn’t want to spend weeks relearning Cisco. Time to tear down the old home network!!

A New Router

My plan is to have 4 “real” networks for my “physical” equipment:

  1. The family’s wireless network – for phones, tablets, game consoles, and tv sticks.
  2. My wired network for my personal workstations and servers.
  3. A VOIP network for POE phones, ATAs, and my PBX.
  4. A server and network lab for me to wreck things.

When I say “real” I really mean “operated by humans” or perhaps “not a Proxmox host”. When I say “physical” I also mean “operated by humans” or perhaps “not a Proxmox host”. At least half of these “real” ports are VLANs, and at least half of these “physical” devices are VMs. In this scenario, “real” and “physical” networks and devices are the ones that I and the family use, compared to the networks that are dedicated to the Proxmox cluster.

The critical distinction is that all of these network segments connect to a different port on the router, and have firewall rules to keep them from connecting to each other. In this scenario, a dumb switch plugged into each port of the router will provide a physically separated network at layer 2 (Ethernet) and a logically separated network at layer 3 (IP). It is here that I have used my first batch of dumb old mini switches:

  1. eth1 – Family Wireless, 192.168.10.0/24
  2. eth2 – Personal Wired, 192.168.11.0/24
  3. eth3 – VOIP, 192.168.12.0/24
  4. eth4 – Lab, 192.168.13.0/24

The family wireless network consists of 2 wireless access points, both with 4 dumb gigabit Ethernet ports:

  1. WAP port 1 -> eth1 on the router, uplink to the Internet
  2. WAP port 2 -> eth0 on the NAS appliance
  3. WAP port 3 -> port 1 on the smart switch
  4. WAP port 4 -> port 1 on the other WAP

So, I had my router set up, and plugging a laptop in to each dumb switch let me pull an IP from the DHCP server for the respective network segment. I was also able to browse the Internet. Awesome. I have managed to convert a big, clunky, error-prone network into four smaller error-prone networks. This is progress?

As far as the family is concerned, eth1 on the router is the network. Wireless access to both the Internet and to the data and media stored on the NAS. If I never plug in the smart switch then only I would notice. I have the WAP’s dumb switch plugged in to the smart switch because I have a media server VM on the Proxmox cluster that I want to put onto the wireless network to stream video to tablets, mobile phones and smart TVs. Because the cluster nodes only have 4 network ports, I need to put multiple network connections on to 1 of those network ports. This is where VLANs come into play. This is also where upgrading my knowledge of routing, switching, and firewalls comes in to play with Proxmox: putting the cluster onto all 4 of my network segments using just one network port from each node.

VLANs: everything you hate about dozens of dumb switches, plus virtualization

With the new router working, it’s time to configure the networks’ core: the smart switch.

VLANs are a great way to divide up a big physical switch into smaller virtual networks. A 24 port switch could be broken down into 4 networks, with a a varying number of ports in each network. You can also put a single switch port onto more than one VLAN. The network traffic gets put into the appropriate virtual network by using tags. You can even put a given port into “all” of the VLANs, this is sometimes referred to as a “trunk.” Trunks are used to connect multiple switches together, passing all tags between them.

Dumb switches can’t tag traffic. So, if you want to mix a smart switch that does VLANs with a dumb switch that doesn’t, you need to make sure that your untagged traffic is going out of the right ports. In the hypothetical 24 port managed switch in the example above, if you put port 2 into VLAN 2, and then plug a dumb switch into port 2, then port 2 needs to know what to do with untagged traffic. Traffic coming out of the dumb switch won’t have tags, and traffic going into to the smart switch will lose its tags. This is the essence of “VID” and “PID/PVID”. A VID is a VLAN ID, PVID is a Port VLAN ID. All the ports on the smart switch need to treat all traffic as tagged, even when it’s not. Untagged traffic needs to be treated differently than tagged traffic, basically meaning that “untagged” is just a special category of “tagged”. The PVID is a kind of “untagged == special tag” way for ports to deal with untagged traffic. This is the exact moment that I developed a migraine.

Star Trek guy with severe head pain.I have done a decent job keeping the family wireless network packets away from everything, and everything away from the family by putting each network segment on its own dumb switch. Now it is time to blur those boundaries a bit by plugging each of those dumb switches into the smart switch. My network is broken into 4 subnets, so my VLANs will break down something like this:

  • VLAN 1 – Family Wireless
  • VLAN 2 – Personal Wired
  • VLAN 3 – VOIP
  • VLAN 4 – Lab

I probably don’t need a separate /24 (class C) network for each VLAN, but I am not very clever and I have zero confidence in my ability to design networks or IP schemes. I know how routing works when you are using /24’s so for my implementation VLAN == /24. Also, as I learned in the Virtual Box lab, network designs get real confusing real fast, so having the VLAN tag roughly correspond to /24 subnet helps me to not go completely insane.

The smart switch is configured by a web interface. This interface has a default IP of 192.168.0.1, so I set a static IP on the Ethernet port of my laptop and logged in. This part of the configuration is important, and it will come into play again later. Once I have all the VLANs set up, I still need to be able to access the switch on this IP address.

I configured the first 4 ports on the switch as access ports or up-links to the dumb switches. Because the dumb switches don’t tag traffic, I needed the uplink ports to treat all “untagged” traffic as tagged to a single VLAN, using the PVID:

  • switch port 1 – VLAN 1, PVID 1
  • switch port 2 – VLAN 2, PVID 2
  • switch port 3 – VLAN 3, PVID 3
  • switch port 4 – VLAN 4, PVID 4

So now, if I change port 5 to VLAN 1 and PVID 1, I can plug in my Windows laptop and pull an IP from the wireless network. Then I can change port 5 to VLAN 2 and PVID 2, and now I can pull an IP from the wired network. Now I need to figure out how to get my Prox cluster nodes to sit on all 4 networks at the same time using a single switch port for each node.

Enter the Management Workstation

Up to this point, I was able to set up my dumb switches and my VLANs with a Windows laptop. I just disabled the WiFi and plugged the Ethernet adapter into the various switches and ports. This was fine for scenarios where one switch port corresponded to just one network segment. But it turns out that Windows can’t do VLANs without proper hardware and software support for the NIC. If you have a VLAN-aware NIC and the Intel or HP enterprise app to configure it, I guess it works fine, but there is no Windows 10 app for the Intel NIC in my crashtop.

In my Virtual Box Proxmox lab, I learned that life is just easier when you have a Linux box dedicated to managing the cluster and testing your network setup, so I decided that before I set up the cluster, I should set up a “Management Workstation.” For the BoxProx lab, I used a Virtual Box VM running a GUI to administer the cluster because I needed a browser on the host only network. Technically, I could have run the management workstation without a GUI and just used SSH tunneling to access the web management interfaces for the Proxmox VMs, but I didn’t want to spend any time doing stupid SSH tricks. I also don’t have the actual hardware cluster running yet, so I need to do this with actual hardware. The hope is that once I get the VLANS and network bridges configured, the workstation will be superfluous. Therefore, the workstation doesn’t have to be powerful at all. Literally any old laptop or desktop that is laying around will do nicely.

My operating system of choice is Turnkey Linux Core. Set up an old desktop on port 5 of the smart switch. For the initial install, I left port 5 configured for VLAN 1 and PVID 1. I was able to pull an IP address from the wireless network, install and update the OS, and configure SSH.

Remote access is important because I can’t sit in my basement all day; Internet access is important because I need to install some network tools.

First step is to get the VLAN tools installed:

apt-get install vlan

Then enable VLAN support in the kernel:

echo 8021q | tee -a /etc/modules

Then add your tagged network interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

Then reboot the machine. I know there is a bunch of crap that you can do to avoid that, but this is the only way I can be sure that it works. I also know that if you name the interface eth0.N you probably don't have to mark the 'vlan-raw-device' but the Debian VLAN tutorial did it so I did it too.

What this does is change the IP of untagged interface eth0 to 192.168.0.10 (remember the IP of the switch from before?) and add eth0.1 (VLAN 1) with an IP of 192.168.1.10 and configured a default gateway and DNS for that interface.

Now, the machine should still be connected to the Internet, and you can modify port 5 on the smart switch to be in VLAN 1 and PVID 1.

If you can ping the IP for the smart switch (192.168.0.1), the IP of something on your wireless network (like an access point) as well as Google's DNS (8.8.8.8) then you are in good shape.

At this point, I left the basement and went upstairs. I connected my laptop to the family wireless network (192.168.1.0/24) to SSH into the workstation. Since I will be making modifications to the smart switch configuration, as well as the management workstation, I decided to configure PuTTy to drop a local port and forward it to 192.168.0.1:80 so that I can access the web interface of the smart switch from my laptop, and the unencrypted HTTP traffic will be secured by the SSH tunnel.

Now I just need to move the Internet access to the 'Lab" VLAN and add the remaining VLANS to /etc/network/interfaces:

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.0.10
    netmask 255.255.255.0

auto eth0.1
iface eth0.1 inet static
    vlan-raw-device eth0
    address 192.168.1.10
    netmask 255.255.255.0

auto eth0.2
iface eth0.2 inet static
    vlan-raw-device eth0
    address 192.168.2.5
    netmask 255.255.255.0

auto eth0.3
iface eth0.3 inet static
    vlan-raw-device eth0
    address 192.168.3.5
    netmask 255.255.255.0

auto eth0.4
iface eth0.4 inet static
    vlan-raw-device eth0
    address 192.168.4.5
    netmask 255.255.255.0
    gateway 192.168.4.1
    dns-nameservers 8.8.8.8 8.8.4.4

The last step is to make sure that smart switch port 5 is part of VLANs 1, 2, 3, and 4, with PVID 1. If all goes well, the workstation can ping the smart switch IP, Google DNS, and servers on all 4 VLANs.

The next step is to use this same network setup for the management NIC on the Proxmox cluster. Using the 4 VLAN interfaces for the network bridges (VMBR1-VMBR4).