If you tried to use Driftsmoke on October 21 and got an error or a maintenance page, this is why. The short version: a critical security fix for PHP, the language our backend runs on, was announced a day before it was released. We decided the safest thing to do was take our services offline until we could apply it. Your account, your saved jobs, and your watches are all fine.
Here is the longer version.
What we were told
On October 20 the PHP project posted an advance notice that a security release was coming on Wednesday, October 21 at 14:00 UTC, which is 7:00 AM Pacific. It would fix two vulnerabilities, both rated 9.2 out of 10 on the CVSS scale. No details were given, just a heads-up to be ready to upgrade. The plan was the obvious one: update the moment the patches were out.
What changed overnight
While we were watching for the release, reports started circulating that security researchers had found the bugs on their own, ahead of the coordinated disclosure, and were passing around proof-of-concept code that could be used to exploit them. Their claims pointed at a specific part of PHP: the feature that tracks file upload progress inside sessions.
We had no way to verify any of this. It was not clear whether they had found the same bugs the PHP team was fixing, whether they fully understood what they had found, or whether they had stumbled onto something new. But the cost of acting on their claims was low, so on the evening of October 20 we turned off that session feature in our PHP configuration and blocked file uploads to our API altogether. Then we kept watching.
In the early hours of October 21, a PHP core developer who had seen the unreleased fixes said in plain terms that the only mitigation was to update PHP. Configuration changes were not going to cut it. That settled it for us. We took the API and the apps offline until the patch could be released and applied, and put up the maintenance page you may have seen.
Why we didn’t just lean on our firewall
Driftsmoke sits behind Cloudflare. They were briefed on these vulnerabilities and pushed emergency rules to catch the known exploit attempts for sites that had not patched yet, on every plan including the free one. We were glad to have that in front of us. But Cloudflare themselves called those rules a stopgap, not a fix, and their CTO said the only way to fully protect a site was to update PHP once it was released. Rules that match known attacks do not match the ones nobody has seen yet. So we still chose to go offline.
What we’re changing
Two things came out of this that we are going to fix.
Only Cloudflare should be able to reach our servers. Right now, someone who knows our server’s address can talk to it directly and skip Cloudflare entirely, which also skips every protection Cloudflare puts in front of us. We are going to lock the servers down so that only Cloudflare’s network can reach them.
We need a way to tell you what is going on. When we took things down, we discovered we had no good way to say so. The website just errored. The app just failed. That is not acceptable, and this page is the first step toward fixing it. It is where we will post what we are building, what is new, and what went wrong when something goes wrong. We will also add proper status messages to the site and the apps so that an outage looks like an outage and not a broken product.
What this means for you
Nothing about your account changed. Your saved jobs, your watches, and your alert settings are exactly as you left them. We took the services down as a precaution, not in response to an attack, and we have seen no sign that anyone got in. Job alerts that would have gone out during the downtime will go out once we are back up.
We would rather eat a few hours of downtime than gamble with your data. Thanks for your patience, and if you have questions, get in touch.
Jason Driftsmoke