Four GitHub Actions jobs were hitting Google Trends every Sunday night, one per client site, each fetching the same keywords — 墾丁包棟, 恆春民宿, 墾丁住宿 — against the same Taiwan region, over the same 18-month window. Four jobs, four trends-state.json files, four chances to get rate-limited. The data in all four was, by construction, identical.
This piece is about the moment I noticed that and what the cleanup actually required.
The pipeline this belongs to
Every B&B in the portfolio gets an automated weekly report. One ingredient in that report is year-over-year search demand for the Kenting/Hengchun tourism market — is the market itself up or down, independent of any one site’s traffic? That context stops the report from misreading a market-wide dip as a site-specific problem.
Google Trends is the source. But Trends rate-limits aggressively, and a 429 during the weekly report run would block the whole report. So the Trends fetch was split into its own workflow that runs Sunday 01:00 Taipei — about eight hours ahead of the Monday report cron — commits a cached JSON, and the report just reads the cache. If Trends rate-limits, the snapshot job exits 0, the old cache survives, and every report still runs. That decoupling was already correct.
What wasn’t correct was the matrix.
The data was never per-site
The snapshot workflow ran a matrix over all four sites. Each site had its own trends.keyword_groups block in site-config.yaml, each fetch wrote sites/<site>/trends-state.json, and the report for each site read its own copy.
The problem: all four sites are in Hengchun. They compete in exactly one market. Their keyword groups were copies of each other — 墾丁包棟, 恆春民宿, 墾丁住宿 — because there is only one set of search terms that describes “someone looking for a Hengchun B&B.” The per-site structure implied the data varied by site. It didn’t. I was paying four times the rate-limit risk to produce four byte-identical files.
The right model is market-level: fetch the Kenting/Hengchun market once, write data/trends-kenting-hengchun.json, and have every site in that market read the same cached file.
What the collapse touched
The refactor was small in lines and wide in surface — the usual shape when you’re correcting a data-ownership mistake rather than adding a feature.
- The Python fetcher gained an
--outputflag so it can write to a shared market path instead of the hardcodedsites/<site>/trends-state.json. The default still writes the per-site path, so the tool stays general — a second market later just adds another step with its own fetch site and output path. - One site (cometrue-bnb) keeps the full
keyword_groupsdefinition and acts as the trigger that drives the fetch. The other three sites’ configs drop their duplicated keyword blocks entirely and keep only adata_pathpointing at the shared file. The render layer readsdata_pathwhen present. - The workflow lost its
matrix— four parallel jobs became one. The four-way fan-out that was fetching the same thing four times is gone.
The asymmetry — one site owns the keyword definition, the rest just reference the cache — is deliberate. It puts the keyword list in exactly one place. When the market’s relevant search terms shift, there’s one config to edit, not four copies to keep in sync. Keeping them in sync was never going to happen reliably; the duplication was a latent drift bug waiting to happen.
Why this matters past the line count
For a single-site project, four-vs-one parallel jobs is noise. For nine B&Bs across five codebases — where each new client could in principle add another Trends fetch — the per-site matrix was a structure that scaled the wrong way. Every site added would have added another redundant hit against a rate-limited API to produce another copy of data that’s the same for everyone in the market.
The market-cache model scales the right way: a new Hengchun B&B reads the existing file and adds zero fetch load. A genuinely new market — a Taichung brand, say — adds exactly one fetch step. The cost now tracks the number of distinct markets, which is what it always should have tracked, instead of the number of sites.
None of this is glamorous. It’s the kind of correction that only shows up when you stop and ask what the data actually is, rather than what the directory structure says it is. The directory said per-site. The data said per-market. The data was right.