Traffic fell like this (indexed to the day before the incident):
| Day | Relative traffic |
|---|---|
| Day 1 (baseline) | 100% |
| Day 2 | ~7% |
| Day 3 | ~3.5% |
| Day 4 | ~1% |
Not an instant outage. A slow rot. That pattern has a name: DNS cache expiry propagating across ISPs at different rates as TTLs expire around the internet.
An A record captures a moment, not a service
The root cause is simple: a Cloudflare DNS A record pointed at an old Fastly IP — one Railway had previously assigned to this service. At some point Railway adjusted its infrastructure and changed IPs. The new IP had the correct TLS certificate and the service was running fine. The old Fastly server no longer recognized the client’s domain and returned HTTP 421 to every incoming connection.
DNS TTL meant cached records kept working. Every ISP, browser, and OS that had cached the old A record continued to resolve correctly until its cache expired. As caches expired in different places over three days, more and more requests hit the wrong server. That’s why it wasn’t a single failure event — it was a three-day gradual death.
The Cloudflare proxy added a diagnostic misdirection layer
Without the proxy, you’d see the Fastly 421 error page directly. The source would be obvious. With Cloudflare in front, the error surfaced as what looked like an SSL problem — the error page came from Cloudflare’s edge, not Fastly. The fact that “the site has a domain and shows SSL” masked the reality that the backend was responding from the wrong server.
The diagnostic path for this class of failure isn’t intuitive. You have to bypass the proxy and test the backend directly to find where the failure actually originates. Starting from the CDN layer and working inward wastes time in the wrong places.
The ad spend kept running
Three days of ads continued burning while the landing page was a Fastly error screen: hundreds of clicks, several thousand NT dollars in spend across the window, zero conversions. This is the direct business cost of a cloud infrastructure IP change — not technical debt, but ad budget already spent delivering visitors to a broken page.
Fix: replace the A record with a CNAME
Deleted the stale A record. Added a CNAME pointing at Railway’s service hostname (DNS only, not proxied through Cloudflare). A CNAME resolves to whatever IP the target hostname currently resolves to. If Railway changes its IP again, the CNAME follows automatically — no manual update required.
The fundamental problem with A records is that they hardcode a point-in-time IP. Cloud infrastructure changes IPs — it’s expected behavior, not an anomaly. A hardcoded A record is deferred risk: you don’t know when it detonates, only that it will.
Monitoring needs to probe below the CDN
The failure signature here — site “appears to exist” but actually serves an error page — is a class that CDN-layer monitoring won’t catch, because the CDN itself is healthy. What’s broken is the route behind it. Availability monitoring needs to probe from a user-side vantage point that bypasses the proxy, or run a separate backend health check that hits the origin directly. Checking CDN status isn’t enough.