I'm curious to hear well-informed reasons from this crowd for why we can trust Tailscale given the non-self-hosted part of the architecture? Does it come down to Tailnet locks [1], not worrying that Tailscale will be compromised, not worrying that your home network is worth compromising, or something else?
Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust".
So while also implementing Tailnet locks and other security measures to constrict traffic flow, I'd also consider going a step further by only permitting server or resource access based on client certificate validation (in other words, a client that's missing a trusted certificate is rejected from even attempting to initiate AuthN); that way even if your Tailscale network is compromised somehow, untrusted clients and endpoints can't make inroads into your infrastructure as easily.
Is that a gigantic PITA to implement? Oh heck, you betcha it is, and I doubt 99% of folks need to go that far with their homelabs or home services. Still, that'd be my approach to zero trust - trusting Tailscale only so far as enabling virtual networking, but not blindly trusting traffic coming over that network at any point.
> Is that a gigantic PITA to implement? Oh heck, you betcha it is
I use my own self-hosted Wireguard VPN server. I agree with a lot of what you were saying about client certificates etc. And I plan to eventually do that sort of thing on some of my services in my own Wireguard VPN too.
But in terms of Tailscale, if you are going to set up all kinds of client certificate things that will take a lot of time and effort, why not self-host Wireguard also?
Setting up a Wireguard server is super simple. The only couple of things that complicate it a tiny bit is opening up a port for it for inbound connections if you host it from your home connection rather than a rented server, and managing the Wireguard public keys that are allowed to connect.
But if you are going to do a whole client certificate setup on top anyway, the work of setting up your own Wireguard VPN is small in comparison.
> But in terms of Tailscale, if you are going to set up all kinds of client certificate things that will take a lot of time and effort, why not self-host Wireguard also?
Already do! I tried Tailscale initially, but ultimately decided to put in the effort of a proper Wireguard setup. It's how my personal devices always get back to my home LAN, and then exit to the internet; it's also how I make sure every DNS lookup hits the Pi-Hole, for domain blocking wherever I am.
I emphatically recommend learning WireGuard (and to a lesser degree, VPN Concentration) when practical and possible. Until then, Tailscale is an excellent product.
No. I was talking specifically about the case where you want to host the Wireguard VPN server at home.
See earlier in the comment where I said:
> opening up a port for it for inbound connections if you host it from your home connection rather than a rented server
Although I can see how it might not be clear that in the end where I’m mentioning CGNAT I am still specifically talking about hosting the VPN server from your home connection.
How is this a good solution, when traffic is decrypted in the cloud, all traffic goes through one node, there is no ACL, key distribution, static IP, …?
I guess I'm not clear what "when traffic is decrypted in the cloud" means but, here's how it works...public traffic comes in on port 80 to the VPS, Wireguard is configured to route it over the VPN to a VM on my home machine. I control the VPS and the peer receiving the traffic.
I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking, especially for businesses, startups, or homelabs where the focus might be elsewhere or specific talent is unavailable. The problem arises when folks don't quite understand why specific decisions are being made, or use the product in nonstandard (or even negative) ways. I've seen stories of folks deploying Tailscale on every machine in their LAN, thinking that secures their traffic; using it to cross boundaries in the firewall or router between secure and insecure VLANs; and using it to connect to servers in lieu of a proper router or firewall with appropriate ACLs.
Tailscale is an excellent piece of software, provided it's implemented in a way to emphasize security, and not weaken it. In OPs case, being used as an accessibility aide to a system that couldn't be secured any other way while preserving external access (in their case due to CGNAT) was an excellent use of Tailscale.
> I do think this simplicity is exactly what contributes to those weird and non-standard configurations.
This is why I am confident I will always have employment in IT. As I make things simpler for others to use, they in turn will find new and innovative ways of making my eyes bleed from cursed workflows that once again require professional intervention for simplicity, efficiency, and security.
> I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking
Tailscale is based on Wire Guard, isn’t it? Now there’s a piece of software that truly made VPNs simple. I have a tunnel back into my LAN by way of an EC2 instance and all it took was two super simple config files on each machine.
Wireguard vastly simplifies the transport level, and attains high performance because it runs in the kernel.
Tailscale simplifies: authentication (including OIDC), authorization (via ACLs), DNS, NAT piercing. All of that is not obvious or easy for someone without deeper expertise.
They have nice clients (e.g. for MacOS, Tizen). Ofc headscale is a thing, but if you have a company, it's also nice to have someone to yell at if your mission-critical tailnet suddenly b0rks.
Imo they don't charge all that much relative to their value, depending on who you're asking.
> Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust".
…which is why I qualified it with the phrase, “extreme interpretation of”, and made sure to encapsulate “Zero trust” in quotes to make it clear I wasn’t being technically literal in my description. Grammar and punctuation matter when you’re deliberately misusing a known term as a metaphor to make a point.
That being said, the core concept of ZTA is that no user or device should be trusted by default. So yes, my statement is still generally correct even if it’s not how the term is often or commonly used.
Not the same. In particular you don’t need to accept software updates from software suppliers and you can also require source code or use open source.
This stuff was obvious and standard in the 80s-2000s. It’s only in the last 15-20 years that it became acceptable to get updates shoved down your throat.
Service providers can cut off your access any day.
Software providers cannot unless you’ve given them a live update channel direct to your env.
I mean, yes? It's why Zero Trust is growing as an operations model. Supply chain attacks, vendor hostility, zero days being hoarded by nations and bad actors for exploit, the list goes on.
You emphatically cannot trust vendors, suppliers, users, software, systems, or governments. Ergo, your infrastructure should be built with an appropriate risk assessment in mind, and have proper safeguards in place where feasible. That's just good OpSec.
Definitely not true. You can audit software (it could be not easy, but ultimately doable) and skip the updates until you have capacity to audit those. You can't audit a third-party service, no matter what you do.
Particularly as it does not include its own PKI, so E2EE is done by MITM your IdP (OICD/SAML etc) and therefore, under court order Tailscale can decrypt your traffic.
We took the opposite approach with NetFoundry. (1) We open sourced the code (https://openziti.io/), (2) we built in PKI with private keys generated at source and destination so that even if traversing NF hosted data plane, we CANNOT decrypt traffic, (3) mTLS everywhere, (4) ability to bring your own PKI, and more.
First of all, a node added to tailnet doesn’t not have the ability to decrypt the traffic in tailnet. All it can do to contact other nodes, decrypt traffic users sent to that hidden node, or modify settings in admin console. Furthermore, with tail lock, the coordination server should not be able to add nodes from outside.
Even if an attacker such as the government runs the coordination and relay servers, and the IdP, they will not be able to decrypt any traffic in tailnet.
The secret keys remain on device, and traffic is end to end encrypted. There is no mechanism in
the client agents to send out the secret keys. The coordination server receives the public keys and metadata.
I see I did have a misunderstanding. I believe there is still the meta data angle, but yes, private keys on endpoints would ensure E2EE. I will update my comment.
> I'd also consider going a step further by only permitting server or resource access based on client certificate validation
This is where I'm the most curious on what Tailscale will do next. So far all their products seem to contrast at the IP level, but for enterprise use cases there's a real need for application level protections as well. Cloudflare Access is a great example of what I mean.
What are the primary downsides of self-hosting this? The top issues that come to mind:
1. Maintaining high availability
2. Dealing with patches/upgrades
But I'm also really curious how likely a self-hosted instance is to be an attack vector potentially more dangerous than using something like Tailscale.
In my experience as a poor sysadmin (as in, bad), you don't /need/ HA for Headscale because the clients are pretty resilient. I've had my instance go down for a little bit and it's fine. Stale and new connections aren't, obviously, but it will work well enough that you won't realize Headscale itself has gone down until a while after it did.
My experience has been the opposite: I have to restart headscale at night because a significant %age of the time when I do, the tailnet goes down. I'd say maybe 30% of the time, maybe more. I'm talking about when I update ACLs and OS updates. I run a single instance, and I will say it's been reliable over ~2 years. For the record, I just rebooted that node for OS updates and the tailnet stayed up.
You own your attack surface at that point. Tailscale/Headscale is a matchmaker and key broker for the most part, the clients almost always (barring NAT issues) connect directly to one another. The normal security considerations apply as with running any service.
I trust Tailscale with my network traffic. I also trust a $50 cheap chinese 10G switch that I bought off amazon with a terrible and surely insecure management interface. Which is to say - I don't, but I don't need to trust it far.
I do have enough trust in their client that's installed on my machine to believe that it's not actively malicious. I do trust that I can find my other devices, and trust tailscale to keep a list of them, and not randomly add other devices that I don't know, but I don't have perfect trust of that. All my internal services are still E2E encrypted over the Wireguard link; They run HTTPS with an internal cert authority. There's not ports open on them that shouldn't be, and while it's possible that one of them still gets popped, it's much less likely.
I don't really understand this though.. The key exchange is perhaps the most important aspect.
Just hypothetically, what if an intelligence service records your encrypted traffic and also happened to get AWS to mitm your communication with the tailscale key distribution server?
Doesn't really matter if most of your traffic doesn't use their infrastructure if the most important parts of it do.
> Using Tailscale introduces a dependency on Tailscale’s security. Using WireGuard directly does not. It is important to note that a device’s private key never leaves the device and thus Tailscale cannot decrypt network traffic. Our client code is open source, so you can confirm that yourself.
My understanding is that (in theory) the only way this is possible is if the attacker introduces a new node and then connected to other nodes that are in the tailnet. What you're suggesting is that a single node that is connected to the other nodes gets compromised, but this isn't possible without already being able to compromise that specific node. Alternatively, if someone hacks Tailscale itself, the only way they could get access to any nodes would be to add their own node, but if you have alerting set up you would know and you could shut down the attacker.
I still find SSH adequate for connecting to a home server remotely. I don't have the CGNAT terrible problem but I also don't do any port forwarding on my home router.
Instead, I have a VM running on a cloud provider that I SSH to from an OpenBSD box inside my home network. The SSH connection establishes a reverse SSH tunnel. This opens a port on the cloud VM to tunnel to my OpenBSD sshd port.
With the reverse proxy to my home OpenBSD box established, I can use the SSH jump box option, -J. I connect to the cloud VM and "jump" through the tunnel to the OpenBSD box at home. You can even specify multiple jumps if I need to connect to another machine in my home.
I can also set up a local tunnel through that jump for things like connecting to my Home Assistant server from my remote laptop or phone.
If I'm understanding correctly, this will break whenever the IP address of your tunnel changes. You'll have to reestablish all of your connections.
My use case for tailscale: have an SSH (or other) connection to my home server while working from home. Drive to a coffee shop, register on their network, and continue using the same connection. (Or hotspot, if I'm somewhere without Wifi.)
The IP address of my server does not change. When at home, the packets do not leave my home network. When out and about, they do.
It's magic to me. I set up a sophisticated (read: overkill) SSH tunneling setup previously, using Match rules in .ssh/config to autodetect the network I was on so that `ssh myserver` would always go via the correct route. But my connections were still interrupted broke when I switched, and I'm not good enough at networking to do any better.
(I guess this is what Wireguard is for? I could access my server via a fixed IP address on my machine that goes to a tun device, and that would send the packets to the actual server if nearby otherwise hand off to the carrier pigeons? Is that what the tailnet is doing? I don't understand how packets get intercepted by tailscaled, though I do see a tailscale0 device. Is that just a vanity license plate version of tun0? Why does `ip route show` give me only routes through my actual devices, then? Never mind, this isn't a helpdesk. I'm just getting old and stupid, I think.)
> If I'm understanding correctly, this will break whenever the IP address of your tunnel changes. You'll have to reestablish all of your connections.
The tunnel is on localhost only. The VM has a static IPv4/IPv6 with DNS.
Connecting the SSH tunnel from my home is stable as well as connecting to the VM remotely.
I do appreciate Tailscale and Wireguard. I was more responding to the fact that I don't have to trust any provider here, other than the one keeping my VM running.
By locked down I mean everything requires authentication (and authorization), everything is containerized, and I have fairly strict firewall defaults.
Let me explain what I mean by low maintenance...
I was a very early containerization adopter and set up a company and also my home network using Docker around 10 years ago. I chose Docker because I thought it was reasonably polished and was the future of deployment. Even though the landscape keeps moving with changes in Kubernetes, Helm, Rancher and stuff like that, the actual Docker part hasn't changed in 10+ years so I haven't had to change my setup for a decade. Low maintenance for me is software that can be left mostly untouched (except for minor updates) for a long time and I judge that based on the project's future, which for me is partly judged from a project's polish.
Every time I tried WireGuard in the past, it didn't seem so polished. I don't want to waste time learning something that could go away. On the other hand, not only did Tailscale look pretty well set up, it was pretty much click and run which means that even if it were to fail, I would have not lost any time learning much about it.
So low maintenance for me is "get the most out of as little work as possible" and choosing Tailscale was the decision to achieve that. So given that I've been using Tailscale for 1.5 years with near 0 amount of configuration and so far, no real downtime, it is adequately low maintenance.
I think it's interesting that they support Kubernetes connections as well, so you can access the control plane, or send data in or out of an environment via Tailscale. I don't have a use for it myself, but it does seem useful.
> Everything on my home network is set up as if it were public-facing.
That's Wireguard, I have the same, just Wireguard + VPS, everything I want available that is. I don't put every PC on my home network on the VPN, I could though, pretty easily.
> Yeah, I don't understand how it is so prevalent in the self-hosted community.
Not just CGNAT but not having _any_ external ports open can be a beautiful thing. I used to have an ssh port (not on the standard 22) and the amount of auth attempts back then was insane. I now have a full firewall zero open ports but, thanks to tailscale, I can still safely access my machines while not being at home with zero unauthorized attempts.
And since I am a security person, I use the tailscale lock feature so not even tailscale themselves can add nodes to my network. Even if they had a breach.
If you're a security person, can you explain why a centralized key exchange server is needed at all? If you care about security you have to verify every nodes key anyway...
Also, it seems their infrastructure runs on AWS, not exactly confidence inspiring from a censorship/privacy risk standpoint.
I think tailscale also doesn't provide transient quantum resistance. Wireguard traffic can be made quantum resistant with a PSK. I fail to see why one would use Tailscale over just wireguard other than for "convenience" reasons which are almost never good reasons if security and privacy also matter. Please correct me if I'm wrong with anything, I'm happy to learn.
> If you're a security person, can you explain why a centralized key exchange server is needed at all? If you care about security you have to verify every nodes key anyway...
I do verify every node’s key. That’s kind of the point of tailscale lock unless I am missing something.
> Also, it seems their infrastructure runs on AWS, not exactly confidence inspiring from a censorship/privacy risk standpoint.
I don’t understand what censorship has to do with a personal home network?
Privacy on the other hand, is fair. For my usecase this is a home network I am not that concerned that they know what devices talk to what devices. Yes they know my ip address but that’s not valuable since it’s all defended by the tailnet lock.
> I fail to see why one would use Tailscale over just wireguard other than for "convenience" reasons which are almost never good reasons if security and privacy also matter. Please correct me if I'm wrong with anything, I'm happy to learn.
Direct access to my network being limited behind tailscale with a requirement to be part of my tailscale network signature satisfies my requirements for no one else’s access to my network at all. And only if I am away from home does any of my traffic pass through a relay.
Tailscale has more device support than any wireguard apps than I know of. I don’t believe wireguard has Apple TV support, but tailscale does.
I am not the only member of my family either, including them in this network with the simplicity of tailscale’s apps is also important.
Wireguard, unlike SSH, behaves like a closed port unless the client successfully authenticates. As far as an unauthenticated client is concerned, you don't have a listening service ("opened port") at all.
I mean, yeah, if you unfortunately have to deal with CGNAT, then you gotta do what you gotta do. But other than that, what's the issue with self-hosting Wireguard?
> But other than that, what's the issue with self-hosting Wireguard?
User simplicity. I am not the only one on my home network which I want to be able to access some parts of the things I build.
Device support. I appreciate that tailscale has gone out of their way to bring tailscale to even more devices than even wireguard supports. Namely apple tv, wireguard does support iOS but doesn't seem to currently support apple tv or maybe just my version of apple tv.
[1]: https://tailscale.com/kb/1226/tailnet-lock