The aggregator, dev-signal, existed before the translation layer. I’d built it several months earlier to track AI/ML developments without depending on algorithmic feeds — 17 curated English sources, all backed by a SQLite database that manages read state rather than relying on any third-party sync. The list reflects where signal actually lives: Andrej Karpathy, Chip Huyen, Sebastian Ruder, Simon Willison, Eugene Yan on the individual side; OpenAI, Google, Meta, Hugging Face, GitHub on the company side; Hacker News and r/MachineLearning for community signal. Not a passive inbox. A deliberate architecture for staying current with the field I’m building in.
The problem that emerged wasn’t access. It was friction. Chinese is my primary language. Reading argumentative English at seven in the morning — before full alertness, in a second language — is a different cognitive mode than reading Chinese prose. Karpathy writes to make an argument. Huyen structures reasoning over evidence. A post like that requires a different kind of attention than one that lists information. I was skimming the posts that deserved a slow read, skipping the technically dense ones entirely. The aggregator was working. I was under-using it.
DeepL was the obvious next layer. I’ve used it manually for years. For nuanced, argumentative prose — the kind that dominates these sources — it’s meaningfully better than Google Translate, specifically on sentences where intent matters as much as word choice. A slightly wrong translation of an argumentative sentence shifts the claim, not just the phrasing. The alternative was LLM-based translation: more flexible, more expensive by an order of magnitude per article, and not meaningfully better for structured, continuous prose. DeepL’s free tier covers 500K characters per month — tight enough at my source count and article frequency that every decision before the API call has budget consequences.
Language detection before spending quota
The first decision was detection, not translation. A handful of my 17 sources occasionally publish in both languages simultaneously. Sending a Chinese article to DeepL wastes quota and produces broken output that can be hard to catch on a quick scan.
I evaluated language detection libraries and decided against them. For a feed that’s English 95%+ of the time with rare exceptions, a Unicode character-ratio heuristic is sufficient and carries zero runtime cost:
export function detectLanguage(text: string): string {
if (!text || text.length < 10) return 'en';
const chineseChars = text.match(/[一-鿿㐀-䶿]/g);
if (chineseChars && chineseChars.length / text.length > 0.3) return 'cmn';
const japaneseChars = text.match(/[-ゟ゠-ヿ]/g);
if (japaneseChars && japaneseChars.length / text.length > 0.2) return 'jpn';
const koreanChars = text.match(/[가-ᄀ-ᇿ]/g);
if (koreanChars && koreanChars.length / text.length > 0.2) return 'kor';
return 'en';
}More than 30% CJK characters means Chinese. Texts under 10 characters fall through to the English default — not enough signal to classify reliably, and a misclassified one-liner is a proportionally expensive mistake against a 500K monthly budget. No external dependencies, no model to download, no accuracy-recall tradeoff to tune. Against these sources, no false positive has been worth investigating.
Ingestion-time, not read-time
Translation runs when an article arrives, not when it loads in the reading interface. Adding API latency to every page view would make the reader sluggish — a problem I didn’t need to create. By the time I open an article, the translated title and summary are already sitting in the same SQLite row.
The deepl-node SDK handles the translation call. At ingestion, the RSS fetcher calls translateArticle(articleId, ['title', 'summary']). Title and summary only — not full content. Full content translation would exhaust quota faster, and the summary is almost always enough to decide whether to read the full English text. The schema stores title_zh, summary_zh, and content_zh alongside their originals in the same row, meaning every read is a single query. No joins, no separate translations table — the only join a translations table would add is overhead for a one-to-one, fixed-target-language relationship.
The FTS5 index covers all four text columns — title, content, title_zh, content_zh — so search works across both languages from day one, without a separate indexing step.
Deduplication that fits the problem
The RSS fetcher calls articleExists(url) before inserting — a SELECT on the url column. If the article is already present, it skips. The articles table has url TEXT NOT NULL UNIQUE as a backstop, so a duplicate that slips past the pre-flight check fails at INSERT rather than creating a duplicate row.
No hash cache. No separate dedup table. No pre-flight race condition — the scheduler is single-process, running hourly via node-cron, so SQLite’s serialized writes mean no concurrent insertion conflicts. I expected to add something more sophisticated once the pipeline had run for a few months. Six months in, the pre-flight SELECT plus the UNIQUE constraint has been sufficient.
Where the glossary problem stays open
Domain terminology is where DeepL falls short in ways I haven’t engineered around. 可觀察性 is the correct translation of “observability” — it’s not what Chinese ML engineers actually say. “Canary deployment” parses correctly in Chinese but reads like something translated rather than said. “Backpressure,” “distributed tracing,” “eventual consistency” — each has a practitioner usage that diverges from the dictionary translation, and the wrong term can shift which concept the reader takes away.
DeepL’s Glossary API is the right architectural direction: upload term pairs, apply them consistently during translation. The integration itself is straightforward. The hard part is the maintenance pipeline. Every new article introduces jargon I haven’t catalogued. A manually seeded glossary starts comprehensive and goes stale as new terminology emerges. An automatically extracted one requires a separate validation step to determine which terms actually need special treatment versus which the literal translation handles correctly. Current workaround: a short manual list where I preserve the English term in parentheses after the Chinese for known problem cases. It doesn’t scale.
For the broad category of what I’m reading — blog posts about AI systems, model behavior, tooling decisions, research retrospectives — DeepL’s output is clean enough that I read the Chinese version without checking the original. For the technically dense posts — Karpathy’s essays, Ruder’s NLP deep-dives — the translation occasionally has the wrong center of gravity: sentences that parse correctly but feel like they were translated rather than written. Semantic content accurate, texture wrong.
That gap is the part of this pipeline that doesn’t have an engineering answer yet. The Glossary API closes it partially. The problem of knowing which terms to put in the glossary, and keeping that list current automatically, stays open.