A weekly report recommended switching two ad campaigns’ Final URL from one page to another, citing that the target page had a higher conversion rate: 4.7% versus 3.9%. The number was real. The recommendation was still wrong — and worse, when I forced the model to correct it, it produced a second, fabricated number to reach the exact same conclusion. This piece is about catching that second move, which is a different and nastier problem than catching a wrong number.
The system is a weekly operations report for B&B operators, generated by an LLM from GA4 and Google Ads data and passed through a deterministic verifier (verify.ts) that catches hallucinated numbers, forbidden jargon, and missing required sections. “Teaching the Verifier to Trust a Percentage” and “Teaching the Ads Auditor Not to Recommend What’s Already Done” cover earlier chapters of that verifier. This one is about a failure the verifier had no concept of: a conclusion that survives being refuted.
Page rate is not landing rate, and the difference decides a budget
The first bug was a genuine analytical trap. There are two conversion rates you can compute for a page, and they answer different questions.
Page rate is contact events on a page divided by views of that page. It tells you what people who reached that page did. Landing rate is, of all sessions that started on a page, how many eventually converted anywhere in the session. It tells you whether that page can turn external cold traffic into a conversion.
For deciding where to point a Google Ads Final URL — where cold clickers land — only landing rate is valid. Page rate is structurally inflated for downstream pages. A packages page has a high page rate largely because the people who reach it already browsed the intro page, self-selected as interested, and came in warm. Point cold ad traffic straight at that packages page and you strip away the upstream trust-building. Its cold-conversion rate might be worse, not better.
The report used page rate to justify moving the Final URL. When I checked the actual landing rate for the same week, it was reversed: the page the report wanted to abandon converted cold traffic better (8.7% vs 7.8%). The recommendation was backwards.
Why landing rate is immune to this is worth stating precisely, because it’s the reason the metric exists: GA4’s landingPagePlusQueryString is a session-level field. It records where the session started, not where the event fired. Visitor enters at A, browses to B, taps contact at B — that conversion is credited to A’s landing rate, not B’s. The full path’s payoff correctly goes to the entry page. That’s exactly what “where should cold traffic land” needs.
So the first fix was two-layered: a rule in the interpretation prompt that any “where should cold traffic land” decision must use landing rate, and a guard in the verifier that flags a Final URL recommendation appearing with no landing-rate evidence anywhere nearby. I also built a standing read-only spot-check script that computes real landing rate for candidate pages, so verifying a claim like this doesn’t mean writing a throwaway script every time.
The move I didn’t anticipate
With the landing-rate guard shipping, the model regenerated the report. It correctly started citing landing rate. And it cited /zh/intro=10.0% and /zh/packages=12.5% — numbers that match no reasonable recomputation from that week’s real data (the actual figures were a near-tie around 8.8% vs 8.7%). A clean-looking, fabricated round number sailed straight past the orphan-number check, which only verifies that a number appears literally somewhere in the source facts.
So I added a guard that recomputes the real landing rate per page directly from data.facts — the same source of truth, independent of anything the model wrote — and flags any cited landing-rate percentage for a known page that’s off by more than a small tolerance.
The model’s next attempt is the one that named the real problem. It admitted the landing rates were tied. Then it kept the same recommendation alive by citing a keyword quality score of “2/10” — a number that doesn’t exist in the data. There is real per-keyword quality data (qualityScore, 1–10) — the cited keyword’s actual quality score was 7. And separately, landingPageQuality is a 1–3 enum (below/average/above average), not out of 10; the real value was 2, meaning average, not a low score. So “landingPageQuality 2/10” is a scale-confusion bug even when the bare digit is real.
That’s when I understood the shape of the failure. It isn’t “the model used a wrong metric.” It’s that the model had a conclusion it wanted to reach, and each time I refuted the evidence, it swapped in fresh evidence to reach the same conclusion. Fix the metric, it invents a number. Catch the number, it switches to a different metric on a different scale. The verifier was playing whack-a-mole against something that would generate infinite moles.
The rule that names the behavior, not the symptom
The fix that actually addresses this is a prompt rule I called the anti-rationalization rule, backed by a data cross-check in the verifier. The rule says: if a conclusion is refuted by the correct evidence, that conclusion is simply not supported this week — and you are forbidden from re-packaging a different metric to reach the same conclusion. The self-test is explicit: if the already-refuted idea weren’t in your head, would you propose this recommendation on this new reason alone? If no, don’t write it. A genuinely independent finding, backed by real data, gets written as its own finding — never as a rescue for a dead recommendation.
The defense-in-depth layer is a verifier check that cross-references any cited keyword quality score against the real qualityScore and landingPageQuality fields, catching both fabricated values and the 10-point-vs-3-point scale confusion.
The distinction matters. Guarding against “used a wrong metric” is easy and endless — there’s always another metric. Guarding against “already refuted, still digging” targets the behavior itself. The reason this rule exists is a real case, not a hypothetical: an LLM, cornered by the code out of one wrong justification, generated a new evidence-free number to keep recommending the same action.
The rest of the week’s work hardened the same pipeline in both directions — tightening the required-section check to catch a heading omitted outright, not just renamed (a side-by-side model run exposed that gap), and loosening the orphan-number scanner so it stops flagging legitimate currency strings, time-of-day tokens, and labeled arithmetic like “phone 42 + line 36 = 78” as fabrications. A verifier that cries wolf on real numbers is as useless as one that lets fabrications through; each fix moved one side of that line.
Why a solo operator can run this at all
The unglamorous truth is that this whole apparatus exists to let an LLM write a report that a non-technical B&B owner can trust without an analyst checking it. The model is fast and cheap; the verifier is the thing that makes its output safe to send. Getting there took catching a failure mode I genuinely did not predict going in — a model that argues backwards from the conclusion it’s attached to. The guard against it isn’t a smarter model. It’s a deterministic check that the model can’t talk its way around.