Securing self-hosted services is exhausting, and I want to go back to SaaS

Also available in: 中文 / English (here!)

(Spoiler: Haven't yet, though.)

Hey, fellow operators! Thanks to the PHP vulnerability that came out recently, I'm wondering how many of you folks have spent the past few days patching infra - or been asked to upgrade to PHP 8?

I've got a pile of self-hosted PHP services over here too (thankfully most of them are just for personal use, so taking them offline for a while is no big deal), and most of them run in Docker containers. Images on Docker Hub were not up-to-date with PHP's fix, and even when the image tags were in place, the images were once not working, so I only finished patching the next day. For some projects I had to bump the base image in a Dockerfile I maintain; some other projects shipped their own new images so I could just use them; for the rest, I patched by updating apt packages (Updates in Sury's repo are impressively timely! Thank you, Sury!).

PHP stuff was finished eventually, but that wasn't the end - Node.js' security releases came out too! After slogging through the fixes and updates, I found out LiteLLM got poisoned again (the second time this year), so I had to spend the night auditing and rotating keys!

I wanna quit! Doing DevOps is killing me! I want to go back to SaaS!

Why are we ops people so exhausted?

You're asking me what's wrong? Your LLM definitely knows about that. I can also ask one for you: LLMs make vulnerability discovery faster, which also means ops has to filter through alerts, judge the impact, track patches, and roll out updates safely far more often. For self-hosted services, fixing a vulnerability is only the starting point: you still have to wait for upstream to ship a release, confirm the image actually works, and then update everything one by one. The faster discovery gets, the less time people have to scramble. ^[1]

Security teams and open source maintainers are already run off their feet! Just look around:

• Back in May, Linus was grumbling that the security mailing list had become "almost entirely unmanageable"; everyone is using LLMs to find the same bugs at the same time.
• Published CVE records in 2026 are running well ahead of 2025. For Q2, 20,709 in 2026 almost doubled last year's 11,701. We're already seeing the CVE ID over 100,000 - that's six digits (Node2Shell HTTP2Shell from the Node.js security release above is tracked as CVE-2026-111497) ^[2]. Although the number of CVEs may not be the most important thing, determining and prioritizing which ones are important has also become a big issue.
• To save maintainers' energy, the kernel docs even spelled out a checklist addressed to LLMs.
• Besides the flood of genuine vulnerability reports, there's of course no shortage of AI slop! curl shut down its bug bounty because of the influx of AI slop, so that trolls wouldn't file fake reports for money.

Security ops can of course use LLMs to fix vulnerabilities quickly too, but just like the counter-terrorists in Counter-Strike have to keep the hostages safe, they also have to keep the service running! The big problem right now is that vulnerability discovery is too fast, and in a lot of open source projects there simply aren't enough maintainers. Trolls generating fake reports with LLMs are, apparently, just adding fuel to the fire.

Under LLM analysis and crawler scans, the internet has already become a dark forest and darker. What can I do? Maybe the only option is to hide.

I'm retreating into a private network!

Actually, even before LLMs took off, people already understood the internet's dark forest theory pretty well. Putting a website behind a CDN like Cloudflare, using Cloudflare Tunnel so the VPS no longer needs to expose inbound ports - they are both good ways to deal with the dark forest. Now that mesh solutions like Tailscale are fairly widespread, people can go one step further and put every service that doesn't need public access entirely inside a VPN.

My Nextcloud and Phorge - they're all perfectly fine sitting in my tailnet, and I could even cut off their outbound traffic entirely. They don't need to be reachable from the internet, right? Even though they're all software written in PHP, nobody else can reach them; if I cut off their outbound traffic, they can't even reach the internet either, which removes most of the worry.

Well, by using Tailscale or Cloudflare Zero Trust, people are still handing the network architecture and trust to someone else at the end of the day. If anyone is worried about the day they "rebel" or get compromised, setting up a WireGuard network (or self-hosting Headscale) might narrow the attack surface further.

What about the public services that everyone needs to use? I mean, people can't force all the users to connect to their intranet over VPN. (MDM: Unless...)

For public services, SaaS might just be easier

Well, let's start with the downsides first: Using SaaS makes the internet centralized. Using SaaS is like voluntarily giving away data control. Using SaaS is like wasting money on unneeded things. But one thing is surely true for me in the 2026 LLM world: LLM content is devouring my energy. Attention and mental wellbeing have never been so valuable to me nowadays. I'd rather spend some money on SaaS for a sound sleep. Since I'm already renting VPSes from clouds, turning into SaaS... might not be that big a leap, I guess.

On the recent wave of vulnerabilities, Vonng wrote another post restating his down-with-the-cloud views a couple of days ago. I think getting the firewalls right and narrowing the attack surface are certainly correct, but that's just security 101. As for air-gapping, it sounds somewhat reasonable, but is probably not helpful for services that are publicly accessible. Backups can indeed be air-gapped (and you should keep an air-gapped copy), but an online database can't be air-gapped. If you air-gap the online database, nobody can use the service!

If air-gapping isn't an option, a self-hosted service on the internet would still require that much work^[3]: security maintenance, software updates, vulnerability patching, backups and so on. Of course I don't need to worry about another tenant escaping to my machine through Copy Fail if I'm doing bare metal, but container escape is only one of a thousand and one kinds of vulnerabilities, and I, as one single human, would need to face all of them. Fortunately Cloudflare sits in front of my website and handles WAF rules; otherwise, if I hadn't even set up a CDN, when a 0-day drops, I'd probably be left with only two options other than staring at the screen waiting for the official fix: "pray I don't get hit" or "shut the service down".

So what's the benefit of a managed SaaS? A good cloud vendor would have people, resources, information, and contracts. Cloud vendors have on-call engineers all over the world who can be on call 24/7; I can't. Cloud vendors have the staff and the compute to do reviews and run tests to make sure a fix actually works; I can't. Cloud vendors get advance notice and maybe even patches from private mailing lists (like Kubernetes' distributors-announce); I can't. And if a cloud vendor's service goes down, they have to give users an explanation and usually publish an RCA detailing exactly what went wrong (plus compensation according to the SLA); I probably can't do that either.

In this PHP incident, Cloudflare had WAF rules up^[4] before PHP even published the fix, so at least it could mitigate things a bit. People on Twitter were spreading the word too: Cloudflare has WAF rules up, don't forget to turn them on! As for customers of Automattic, the company behind WordPress.com, their lives are much easier: Automattic patched it for them and they can just sit back and relax. Me? I have to spend the whole night staring at Docker Hub and editing Dockerfiles one by one! I mean, yes, I could automate this with or without LLM, but I'm not running an enterprise that has a CMDB! I'm just one person with several git repos of Docker compose files.

Of course, just make sure it's a trustworthy cloud vendor. They may make small mistakes from time to time, but don't choose one like the Indian CA eMudhra, who used the number 4 as a private key :-)

So, what did I actually do?

I haven't moved everything onto SaaS yet, but I did replace almost all PHP software on my side:

• I replaced NextCloud with the CalDav server project Rustical (it supports OIDC for my Pocket ID instance).
  • I also use NextCloud to store some files, but since I'm only accessing it via rclone I think I can move them onto Backblaze.
    • I challenged myself (and my agent) to deploy oCIS but found it still fairly complicated and gave up. ^[5]
• I put a notice saying my Private Bin service may sunset in a year. I think I'll really take that down then.
• I'm thinking of moving my web analytics tool Matomo instance away (it's used on this page). Still evaluating Plausible; Umami only supports server-side tracking in a very weird way (e.g. logging user IP addresses via X-Forwarded-For).
• I also have a Phorge (a Phabricator fork) instance for personal ticket tracking (inspired by delphij), but I'm not sure what to do with this at present. I moved from Redmine due to my love with Phorge's interesting user experience ^[6], but I haven't found an alternative yet.