Finally getting my network right

I bought in on the Amazon Alexa ecosystem for my home automation a few years ago for several reasons.  While I'm really not a huge fan of Amazon as a company, of the choices out there, Alexa definitely gives you the most options and has a lot of lower cost options than the other Home Automation platforms out there - namely Google Home and Apple Homekit.   Surely, the lower cost of devices comes at a price - namely, your data.  I'm not sure if this has changed in the years since I first jumped onto the smart home bandwagon, but at least within the Apple ecosystem, the one and only smart lighting solution was Philips Hue bulbs which are still insanely expensive and require a controller hub.  Meanwhile, there are tons of inexpensive options for Alexa, but they all tend to rely on some questionable Chinese apps like Tuya/Smart Life, and who knows what kind of phone home/data scraping these cheap bulbs are doing.

Where this whole project started was when my trusty old Google Nest Wifi APs started dying on me, which was wreaking havoc on my smart home stuff.   If you've ever dealt with that, you know just how irritating it is when your smart home devices start dropping off the network.  Flickering lights going back into setup mode became a near daily irritation, and Alexa would sometimes just time out and say something very helpful like "Hmm... Something went wrong".  So it was just time to upgrade.  I ended up going with a pair of TP Link Omada Wifi APs and for the most part I'm quite happy with the purchase.  The new wireless AX is a nice bump in peformance compared to the aging wireless AC Nest access points, but the biggest benefit that the Omadas access points offer is the fact that you can set up multiple SSIDs on separate VLANs, which made it a lot easier for me to cordon my smart home devices off from my main network using a combination of DNS, DHCP and firewall rules.

Given that OPNsense is first and foremost a firewall, I'm always a bit suprised at how relatively little information is available for doing some basic things like setting up a "guest" network.  One would think that creating a VLAN with Internet access and nothing else would be a pretty common goal but the process really is not very well documented at all.  I'm by no means an expert, and I could certainly be going about this wrong, but this is what ended up working for me.  I partly based my setup on the docs here for creating a captive portal (like a coffee shop guest network).  Some of the steps like accessing the captive portal itself are irrelevant to my needs, but most of the firewall rules seem to work.

Network Setup

First thing's first.  I had to create a VLAN specifically for my IoT devices which I named "IDIoT", in my case, VLAN 80, which has the subnet 192.168.80.0/24.  I debated on whether I should use a larger subnet, but as it stands now, between all of my smart home devices including bulbs, switches, LED strips, Amazon Echos, smart thermostats, door locks, Ring doorbell and all my streaming boxes, I'm still only using about 100 IPs, so I have plenty of room to grow.  I'm no networking expert, but I think I could safely use up to a /21 subnet mask (255.255.248.0 in CIDR notation) without stepping on IPs in the next VLAN, VLAN 90, which uses 192.168.90.0/24.  A /21 would give me up to 2048 IPs, or 8 x /24 subnets of 256 IPs, from 192.168.80.0 - 192.168.87.255.  Not that I need 2000+ IPs, or even have network infrastructure that could support that many clients.

Here is a somewhat in depth description of VLAN 80 and it's behavior, and how it differs from the rest of my network.  For most of the other VLANs on my network, I have a simple IPV4 allow any rule, so clients can get out to the internet and can generally communicate with one another across VLANs.  VLAN 80 on the other hand has much more restrictive firewall rules.  First, I created an alias called 'cudanet_vlans' which contains a list of all of the subnets in my network except for VLAN 80.

Firewall Rules

Then with my alias in place, I created the following firewall rules for the IDIoT VLAN - the order does matter.  Firewall rules on OPNsense generally go from most restrictive to least:

1. Allow the IDIoT net to access the DNS forwarder (not sure if this is necessary in my setup since the DHCP segment is pointed directly at Google DNS)
2. Block the IDIoT net from accessing the other VLANs via the alias
3. Block access to the firewall itself from clients on the IDIoT VLAN
4. Allow access to the Internet from the IDIoT VLAN

I really don't know if this is the best way to set up the firewall rules, but effectively I've accomplished what I was after - clients on the IDIoT VLAN can get out to the Internet but cannot contact anything inside my network.  

Avahi

Anything on my network cannot contact anything on VLAN 80 and vice versa - with one notable exception; I installed the mDNS (aka; Avahi/Bonjour) plugin.  What the plugin does is it rebroadcasts mDNS across subnets so that for instance, Airplay from my iPhone on my regular wifi network to the Apple TV on the IDIoT VLAN works as expected.  I honestly don't fully understand how that works or why mDNS doesn't need special firewall rules to work, but it just does with the plugin installed.

Wireless

Now let's go over how the wireless networks were set up.  I have two TPLink Omada access points.  Anyone who has worked with smart home devices is probably aware that the vast majority of smart devices can only use 2.4Ghz networks.  For maximum compatibility, I've created a 2.4Ghz SSID bridged to VLAN 80 on each of my access points, and dropped security down from WPA3 to WPA2.  Generally speaking, smart devices connect very quickly with a lot less timeouts and errors like I would see when I was running a mixed 2.4/5Ghz network with stronger encryption.

I also created an IDIoT-5Ghz SSID as well, specifically for streaming devices like Apple TVs and Rokus to connect to.

One other thing to point out - I toggled the 'Guest Network' option on the VLAN 80 SSIDs.  This further tightens down security on this VLAN.  In theory, this should do the same thing as what I'm accomplishing with the firewall rules on OPNsense, but also restricts communication within VLAN 80 even further by preventing peer devices from communicating with each other as well.  Now, I'm not entirely certain this is the best idea and I may end up needing to disable this feature if it ends up causing problems, but I'm going to try it out for a while and see how it goes.  For now, things seem to be working as intended.

My hunch that enabling the 'Guest Network' feature did end up causing problems.  Specifically, I could not access either Plex or Jellyfin as they would be routed through the WAN interface in this instance, which even if it worked properly is not ideal.  Ideally, you want to be able to hit the server IP (or in my case, the service load balancer VIP) so that you can get direct play without having to go out over the WAN.  

Firewall Rules Cont'd.

After disabling the Guest Network feature on the APs and adding another firewall rule to allow traffic from VLAN 80 directly to Jellyfin, I was able to get it to  work - albeit without SSL, which is kind of annoying.  The only way to get SSL working properly on VLAN 80 would be to use a local DNS server, which in my case would require yet another instance of something like Pihole, which although trivial to set up, is not what I want to do.  I want VLAN 80 to just use Google DNS and sidestep my local DNS entirely to avoid problems with streaming services and such.  The other option would be to create a group on my existing Pihole instances for my streaming devices (eg; Apple TVs, Rokus and possibly Amazon Echo devices) with ad blocking disabled, but I don't want to manage them like pets, I want to manage them like cattle.  So for now, hitting Jellyfin without SSL enabled is an annoyance I'll just have to live with.  To get direct access to either Jellyfin or Plex, create one or more firewall rules like this:

Action: Pass
Interface: IDIoT
Direction: In
TCP/IP Version: IPv4 (I don't use IPv6)
Protocol: TCP
Source: IDIoT Net
Destination: Single Host or Network
   Destination: 192.168.90.203/24 (this is the Jellyfin service VIP)
Destination port range: 
   from (other) 8096 
   to (other) 8096
Description: Allow access from IDIoT to Jellyfin

Keep in mind - the order of the firewall rules does matter.  With the appropriate firewall rules in place, we can check if we have access to Jellyfin or Plex from VLAN 80

As we can see, it works.  One thing to note - specifically on Apple TV, there is no way to set a local IP address for Plex, so in this case all traffic will go out the WAN.  I'm finding less and less reasons to stick with Plex, and that seems like a pretty compelling argument in favor of just dumping Plex and going with Jellyfin.  But for the most part, there's no harm in running both.  They access the exact same movie library via an NFS share on my TrueNAS server and since both Plex and Jellyfin store their metadata and databases separately from the media library itself, there's no harm in providing access to my movies via both services.

Ad Blocking

This is one of the hardest things to get right in a network, because if done wrong you can break a lot of things.  Due to some of the limitations with my previous network setup, I had to forego ad blocking because it was an "all or nothing" kind of thing until I had the ability to segregate select devices to their own wireless network(s).  I was using Pihole in the past to set up whole network ad blocking, but it ended up having the unintended side effect of breaking some streaming services - Hulu and Paramount Plus were particularly problematic with Pihole turned on.  Having my IoT devices on a separate VLAN means that I can specify alternate settings for clients on that network than I do for the rest of my network.  VLAN 80 is configured to use Google DNS (8.8.8.8 and 8.8.4.4), completely side stepping internal DNS, which provides another layer of added security and gives clients on VLAN 80 a transparent path to the Internet.

And then finally, I have two Pihole instances running, one on each of my Openshift clusters.  I configured OPNsense to use them as primary and secondary DNS, and in turn, Pihole is configured to use internal DNS (dnsmasq on OPNsense) for local resolution and then hand everything else off to Google DNS.

This is the block list I've added to both instances of Pihole

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
http://adblock.gardar.net/is.abp.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
https://v.firebog.net/hosts/static/w3kbl.txt
https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt
https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt
https://raw.githubusercontent.com/vokins/yhosts/master/hosts
https://raw.githubusercontent.com/RooneyMcNibNug/pihole-stuff/master/SNAFU.txt
https://v.firebog.net/hosts/BillStearns.txt
https://www.joewein.net/dl/bl/dom-bl-base.txt
https://v.firebog.net/hosts/Admiral.txt
https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
https://v.firebog.net/hosts/Easylist.txt
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
https://raw.githubusercontent.com/jdlingyu/ad-wars/master/hosts
https://v.firebog.net/hosts/Easyprivacy.txt
https://v.firebog.net/hosts/Prigent-Ads.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://v.firebog.net/hosts/Prigent-Crypto.txt
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
https://phishing.army/download/phishing_army_blocklist_extended.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
https://v.firebog.net/hosts/Shalla-mal.txt
https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
https://urlhaus.abuse.ch/downloads/hostfile/
https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
hostfiles.frogeye.fr/multiparty-trackers-hosts.txt
https://winhelp2002.mvps.org/hosts.txt
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&useip=0.0.0.0

Running a quick test on canyoublockit.com produces generally good results.  I don't block 100% of all ads but I block enough that it makes browsing the Internet a lot less terrible

Wrapping Up

In conclusion, I finally have all the questionable smart devices on my network segregated into their own VLAN where they can't snoop around the rest of my network and send my data off to who knows where, and I finally have ad blocking working without disrupting my streaming services.  I just wish I'd done it sooner.

One more layer of tinfoil to add to your hat is to use a throwaway account (or better yet, multiple throwaway accounts with random strong passwords) to register your questionable smart devices to their cloud services.  I wish I could do the same for my Echo devices, but unfortunately I have streaming services and such that I pay for that are tied to my real email (and perhaps more importantly, my credit card).  Same goes for my Apple TVs and Rokus, but I suppose that's just a necessary evil I'll have to live with.

Going Further

On the subject of tinfoil hats, if you're really paranoid about data privacy, and even  with your smart home devices VLAN'd and firewalled down into a dark hole, I came across the Tasmota project which effectively is an open source replacement firmware for things like smart bulbs, and it's largely compatible with most bulbs that use the Tuya/Smart Life app - so... basically all of the smart lighting in my house.  Hooray!  Now, I don't really feel up to the task of manually reflashing firmware on ~100 light bulbs, and it seems as though if the firmware on your bulbs has been updated beyond a certain point, reflashing them is not possible, but it's certainly something that has piqued my interest and I may take look at flashing some of my bulbs, and then managing them with the Home Assistant, which has an Alexa skill and can provide some advanced features like Philips Hue emulation and can even work offline when the Internet is out - which would be nice.  I've only had one Internet outage since I moved into my new house last year, but it wreaked havoc on my smart devices.  I'm certainly interested in testing this out, but like I said, manually flashing firmware on dozens of devices and setting up my own self hosted management seems a little daunting, but... I do like a challenge, almost as much as I like my privacy.