Virtualized OPNsense, Router-on-a-Stick, and Layer 2 Troubleshooting

shaikhnabeel2706@gmail.com (Nabeel) — Fri, 27 Mar 2026 00:00:00 +0000

## Section 1 — The Itch

For a solid year, I was on a nationwide ISP that handed me a router with exactly two settings accessible: change the LAN IP or disable DHCP. Truly generous of them. It was a locked box. I couldn’t even turn off its Wi-Fi to use my own. You get what you get. That was the year I decided local providers exist for a reason.

I switched to a local one who just handed off a pure ethernet connection from their fiber ONT. I got to plug in my own router, a TP-Link, nothing fancy, but mine. It worked. Still, there was not much to customize. I had been running a homelab for a while by then and had my addressing laid out the way I like it. I used a /16 subnet carved up logically. I used .10.x for physical machines, .20.x for VMs, and so on. The router didn’t care about any of that. It just happily handed out addresses with no regard for my schemes. I opened Fing one day to do a network scan, and I don’t think it ever finished. It was pinging absolutely nothing for most of the address range and wasting time waiting for responses that would never reply. I am honestly not sure it has finished to this day.

The obvious next move was OpenWrt. Custom firmware, full control, free. Except I checked the supported device list and my two routers weren’t on it. Not an edge case, not limited support. Just not there. That is the reality of buying consumer hardware in India; the OpenWrt coverage is patchy at best.

So that was that. I started looking at managed switches.

Side note on that: this is actually my second switch. When I was first setting up the homelab, I knew I wanted VLANs. I did not, however, fully understand the difference between managed and unmanaged switches. I ended up with a TP-Link LiteWave LS108G, a cheap, no-frills switch that does exactly one thing: switching. No VLANs, no configuration, just packets going where packets go.

In January, I finally pulled the trigger on a TP-Link TL-SG108E. This is an 8-port managed switch with web-based VLAN configuration, a heavy metal shell, and dual link speed indicator lights. Those dual lights immediately told me I had been running one of my Proxmox nodes on a 10/100 cable. Small win. And that purchase is what kicked off everything that follows.

## Section 2 — The Plan

So the search began. Dedicated router boxes like the cheap N150 units are not available in India. Enterprise gear is expensive, bulky, and overkill. The RPi5 came to mind briefly but it was not the right tool for this. The used office hardware market, the classic homelab goldmine, has gotten out of hand; an i3 SFF with 8GB RAM now goes for 8,000 INR minimum. New hardware in this economy? Please.

What I did have was a ThinkCentre Mini PC with an i5 and 16GB RAM, already running Proxmox. I have been using Proxmox for years on my other nodes (hence PVE1 and PVE2). It is my default OS for anything that needs to run VMs. Virtualizing a router on it was a natural fit, and it was more than capable for the job.

The only real constraint was having one physical NIC. The proper way to virtualize a router is to pass through a dedicated NIC, ideally a dual or quad-port card, entirely to the firewall VM. This leaves the host its own separate interface. If you only have two ports, you are already stretching it: one goes to WAN, and the LAN NIC ends up shared between the host and the VM. With one NIC, that whole premise collapses, which is exactly where Router-on-a-Stick comes in.

Router-on-a-Stick came up while I was researching. It is the idea that a single physical interface can carry multiple isolated networks simultaneously using 802.1Q VLAN tagging. Packets are stamped with a VLAN ID that tells every device on the wire which network they belong to, and the managed switch handles the segregation at Layer 2. One wire, many networks, cleanly separated. For my 300 Mbps symmetrical connection, there is no bottleneck concern at all.

The flow ends up looking like a chain: one physical NIC splits into a host management interface and a trunk to the OPNsense VM, which then presents two virtual interfaces of its own: one for WAN and one for LAN.

My original idea was actually a dedicated WAN NIC via an M.2 A+E key adapter. Those slots on Mini PCs are usually used for Wi-Fi cards, but you can drop in a 2.5G ethernet adapter and get a proper second NIC. Unsurprisingly, they are almost impossible to source in India. I found exactly two sites carrying them. That is still the plan eventually, but the current setup works flawlessly in the meantime.

OPNsense was the natural choice for the firewall. It is open source, actively maintained, and features a UI that actually exposes the controls you want without requiring a doctorate to navigate.

## Section 3 — The Build

I had set up VLANs before in a previous project, but nothing at this level of complexity. A managed switch, a single-NIC router-on-a-stick topology, and ISP quirks thrown in meant this was a different beast entirely. I knew I needed to plan the schema carefully before touching a single config page.

I used a /16 subnet, and that gives you 65,536 IP addresses in a single broadcast domain. Whenever a device needs to find another, like a simple ARP request asking “who has 10.0.0.5?”, it yells that question to every single device on the wire. With a /16 and no VLANs, there are no walls. Every physical machine, VM, and smart bulb hears the background noise.

I had an AI assistant open during the planning phase, mostly as a sounding board to sanity-check the schema before I touched a single configuration page. We landed on a clean, segmented layout:

VLAN 10 (NET_INFRA): Bare metal and hypervisor management.
VLAN 20 (NET_VMS): Standard server and container workloads.
VLAN 25 (NET_DESKTOP): Secured, trusted workstations.
VLAN 30 (NET_WIFI): Since my repurposed TP-Link router does not support VLAN tagging for individual SSIDs, the entire Wi-Fi access point gets dumped into this isolated network.
VLAN 98 (NET_BLACKHOLE): The zero-trust sinkhole for any unused physical ports on the switch. It doesn’t actually exist in OPNsense — no DHCP, no gateway. If a rogue device gets plugged into a port with this VLAN ID, the link lights turn on, but nothing happens. It’s like plugging into an ethernet wall jack that was never wired up. Cut off at Layer 2 with zero network access.
VLAN 99 (NET_WAN): The isolated transport layer for raw ISP fiber traffic.

Up on the Proxmox host, I set the virtual bridge (vmbr0) to be VLAN-aware. This allows the hypervisor to read the 802.1Q tags and forward frames to the correct virtual machines. But it is not just a software toggle; you have to account for physical constraints.

Standard Ethernet frames carry an MTU of 1500 bytes for the payload, plus 18 bytes of Layer 2 header and trailer, totaling 1518 bytes. A VLAN (802.1Q) tag adds 4 bytes to the Ethernet frame, making the total frame size 1522 bytes for a 1500-byte payload. MTU settings usually refer to the Layer 3 payload, but interfaces enforce Layer 2 frame limits. If the interface doesn’t have proper “VLAN-size” support and enforces a strict 1518-byte limit, the hardware just drops the frame as oversized.

For the WAN connection, instead of dealing with VLAN configuration inside the OPNsense UI, I tagged VLAN 99 directly on the virtual NIC inside Proxmox. This effectively creates an invisible, hardware-isolated tunnel straight from the physical switch port to the virtual firewall.

## Section 4 — The Bugs

### The Sequencing Lockout and The Mobile Console

Before I even hit the gateway issues, I managed to completely lock myself out of my own infrastructure. I made a classic sequencing error: I configured the VLANs on the physical switch before I had set up the corresponding VLAN interfaces and DHCP servers in OPNsense.

The moment I hit “Apply” on the switch, my desktop lost its IP. No DHCP. No route to Proxmox. No route to the switch. And because I had already aggressively sinkholed all the empty physical ports into VLAN 98, I couldn’t even plug a laptop directly into the switch to bypass it. I had engineered a perfect, impenetrable fortress around the new VLAN layout, and I was stuck inside it with a broken path from the desktop — not locked out of a working network, but unable to reach the gear I needed to fix from the machine I normally use.

At this point the uplink was still going through my old Wi-Fi router, not straight into the ONT. That is why the phone still had internet over Wi-Fi, and why I could still reach the Proxmox console at all; the house link was intact, only the desktop’s path through the new switch topology was dead.

But I remembered a trick from a past project. If you have console access to the firewall, you can temporarily disable the packet filter entirely, which allows you to access the web GUI via the external IP.

At the time, my phone was connected to my old Wi-Fi router. What followed was one of the most annoying troubleshooting sessions of my life. I had to log into the Proxmox web interface on my phone’s browser, open the clunky HTML5 VNC console for the OPNsense VM, navigate the shell menu on a touch screen, and disable the firewall.

I got in. But every time you apply a configuration change in the OPNsense web GUI, it automatically restarts the firewall service and re-applies the block rules. This meant I had to repeat the mobile VNC shell dance to kill the firewall after every single change. I had to configure all 6 VLANs, the DHCP server for VLAN 25, and all the associated firewall rules entirely from my phone. I even edited the switch config on the phone as well. It was an excruciating, tiny screen nightmare, but it worked. It saved me from having to factory reset the switch and start from scratch.

### The MAC Address Conflict

Spoofing the MAC address of the ISP-provided router is one of the most common techniques to bypass ISP hardware whitelists, so I went with it.

At this stage, my old TP-Link Archer C6 was still connected directly to the ONT and handling the active PPPoE session. My plan was to set up OPNsense with the cloned MAC, test it by pulling a local DHCP IP from the Wi-Fi router, and then physically swap the cables.

I flipped the router over and read the MAC off the sticker, plugged it into the OPNsense WAN interface, and fired it up. The interface couldn’t pull an IP.

Here is what happened: the MAC address printed on the sticker was the LAN MAC of the router, the address ending in :10. The WAN MAC, ending in :11, I found later by logging into the Wi-Fi router’s admin page — not from the sticker. Because OPNsense was plugged into that exact router to test DHCP, I had engineered a Layer 2 conflict. Two distinct devices were claiming the exact same hardware address on the same network. The switch’s MAC address table was thrashing, and OPNsense was left starving for an IP.

The fix was straightforward once I spotted it: the WAN MAC is simply the LAN MAC incremented by one, ending in :11. I updated the spoofed address, and OPNsense immediately pulled a local IP from the router’s DHCP. Test passed. I switched the WAN interface over to PPPoE, unplugged from the Wi-Fi router, and patched the switch directly into the wall jack leading to the ONT.

### The Unplugged ONT

Throughout this troubleshooting loop, a standard debug step suggested by Gemini was to power cycle the fiber ONT after every major configuration change to flush the ISP’s stale session. Because my homelab and the ONT are in two different rooms separated by a wall, this meant getting up, walking over, switching it off, waiting, switching it back on, and walking back to the terminal. I had done this dance several times already.

On what was supposed to be the final attempt, I walked over, switched the ONT off, and just walked back to my desk. I completely forgot to turn it back on.

Ten minutes went by. The logs were showing connection timeouts. I admitted temporary defeat, grabbed my old Wi-Fi router, and plugged it into the homelab side wall jack just to get the house back online. I opened the router’s dashboard. It read: WAN Unplugged.

I walked back into the other room. The ONT was just sitting there, completely powered off. I flipped the switch, walked back, swapped the cable back to the managed switch, and it tried to establish a PPPoE session. More on that next.

### The MTU Trap

Once the PPPoE connection was finally dialing out, I hit a wall of MESSAGE TOO LONG errors in the system logs. Internet traffic was completely stalling.

This goes back to the MTU math. PPPoE adds another 8 bytes of overhead on top of the 4-byte VLAN tag we already had. Worth noting: how these stack depends on where the VLAN tagging is applied, whether at the host, the switch, or the guest. In my setup, both were in play, and the combined overhead was enough to push frames past what the virtual bridge would pass. I fixed this by clamping the OPNsense WAN MTU down to 1484. The textbook PPPoE value is 1492, but 1484 was the number that actually cleared the errors in my environment. More conservative than strictly necessary, but it fit the actual path and the logs stopped complaining.

### The Gateway Blackhole

Getting an IP was only half the battle. OPNsense immediately marked the WAN gateway as “Defunct” with 100% packet loss, dropping my default route. I had an IP, but no internet.

This turned out to be an ISP quirk. OPNsense uses a service called dpinger to monitor gateway health via ICMP echo requests. My ISP silently drops all ICMP ping requests at their edge, so OPNsense assumed the connection was dead. The fix was a single checkbox: System -> Gateways -> Configuration then “Disable Gateway Monitoring.” This forces the routing table to stay active regardless of ping status.

### The VLAN 1 Trap and The Management Lockout

With internet finally flowing, I went to log into my switch’s web GUI from my desktop on VLAN 25. The page timed out.

There were two problems. First, OPNsense’s default LAN interface was still on its factory 192.168.1.1/24 assignment. Second, even with a permissive firewall rule, OPNsense had no route to the switch’s 172.16.0.x network. A firewall rule permits traffic; it doesn’t conjure routes out of thin air.

The instinct here, and a trap a lot of people fall into, is to create a VLAN interface with ID 1 in OPNsense to bridge the gap. The problem is that VLAN 1 is typically the native or untagged VLAN on most switches; it expects untagged frames. If you explicitly create a vlan01 interface, the firewall inserts a Tag=1 header. The switch receives a tagged frame on a network it expects untagged, gets confused, and silently drops the packet. The exact behavior is switch-dependent, but the native versus tagged mismatch will bite you one way or another.

The fix was a Parent Interface Bridge. OPNsense’s default LAN interface was already there from the factory image; I had not created a new one. I simply changed its IP address from the default 192.168.1.1/24 to 172.16.0.1/24. In Proxmox, the raw virtual NIC acts as the parent to all tagged VLAN sub-interfaces. Any traffic sent out of this parent interface is inherently untagged. With that address on the existing LAN interface, OPNsense could natively route traffic as untagged frames directly to the switch.

### Don’t Just Allow All

Allowing routing between VLAN 25 and the native management network required a firewall rule. The lazy approach is an “Allow All” rule, but if a device on VLAN 25 got compromised, that’s a wide open door into the infrastructure layer.

Instead I got surgical about it. I created a Host Alias in OPNsense named SW_CORE_MGMT, hardcoded strictly to the switch’s IP (172.16.0.4). Stateful firewalls apply rules on the ingress interface, so the rule sits on VLAN 25 where the traffic originates:

PASS | Protocol: TCP | Source: VLAN 25 net | Destination: SW_CORE_MGMT | Port: HTTP (80)

My desktop could reach the switch GUI. Everything else on the infrastructure layer was mathematically unreachable.

## Section 5 — The Result

So here we are. A few days into running a custom OPNsense router, and I can report that the migration was surprisingly non-disastrous. The internet works. The family has not disowned me. Mostly.

In day-to-day usage, nothing feels different, which is exactly the point. The packets move, and nobody notices the plumbing. But the small wins add up. Fing actually finishes a network scan now. My VMs no longer clutter the device list alongside my phone and my family’s laptops. I can finally see exactly what is connected to the Wi-Fi AP, cleanly separated from the infrastructure layer. The network just quietly does what I always wanted it to do.

This project also shook the dust off some VLAN knowledge I had not touched in a few years. It came back faster than expected, and filling in the gaps was genuinely satisfying.

If I am being honest about why I enjoyed this so much: it pulls you completely out of your head. After months of grinding through pure code, pipelines, and heavy deployments, picking up a networking challenge felt like switching to a different muscle group entirely. I thrive on problems that are convoluted enough to require actual thinking, and this had just enough of that. The planning phase especially, sitting down and architecting the VLAN schema before touching a single config page, was the kind of focused work that is hard to find elsewhere.

And then there are the dashboards. Any engineer knows the feeling: a screen full of live graphs, counters, and connection states, with every knob visible and tweakable. I could watch the OPNsense dashboard for an embarrassing amount of time. It is the kind of visibility that makes you realize how blind you were flying before. I can see traffic patterns, apply QoS, and most importantly, keep my test workloads completely isolated. The house’s internet is no longer at the mercy of whatever ISO I am pulling into a VM at 11 PM.

Speaking of the house’s internet, I did briefly cut it during the migration. Right after a long day of work. They were not thrilled. I survived.

The other rabbit hole I didn’t anticipate was the OPNsense plugin ecosystem. I opened the community packages list expecting a handful of basic utilities and found myself staring at a wall of genuinely powerful tools. Tailscale integration, performance monitoring, automated config backups straight to a Git repository; that last one alone has me unreasonably excited. I hadn’t planned on running much beyond the base firewall, but future me is going to be very busy.

One last thing worth saying: I had an AI assistant open for a good chunk of this build. It parsed logs, sanity-checked configs, and acted as a sounding board. It helped. But an LLM is only as useful as the person wielding it. You don’t need to be a senior network architect, but a fair working knowledge of virtualisation, Linux, and basic routing goes a long way before you attempt something like this.

I started with partial knowledge myself, and the gaps filled in as I went. But the project taught me more than I expected precisely because I wasn’t just blindly copying commands and clicking through guides. When you do that and hit a wall, and you will hit a wall, you have absolutely nothing to fall back on. Knowing where you went wrong and why is worth more than whatever you end up building. The infrastructure will change. The understanding stays.

To any homelabber reading this: build it. Don’t do it just to put it on your resume. Do it because you can spend hours pulling one thread, watching how changing a single VLAN tag breaks something three layers down, and then fixing it. That loop is deeply satisfying.

There is more coming. I have a few projects sitting in the bag that tie back into this infrastructure, and this network rebuild was partly laying the groundwork for them. More on that later.

And that M.2 A+E 2.5G adapter is still on the shopping list. At 2,199 INR, one day.

Router-on-a-Stick on Nabeels blog