Deduplicating an automated news pipeline — title matching isn't enough

When I started building TechPluse's dedup step, the obvious first pass was string comparison on headlines. It lasted about a day before it was obviously wrong.

The same story shows up across sources with completely different phrasing. "OpenAI ships new reasoning model" and "New o-series model launches from OpenAI" are the same event described by two editors with two different habits. A string match — even a fuzzy one — misses that pair every time, and a pipeline that can't dedup those two headlines will happily publish both as if they were independent news.

What actually works

Semantic comparison, not lexical. Embed the candidate item and compare it against a rolling window of recently stored items using cosine similarity. If something scores above a threshold against an existing entry, it's a duplicate — even if not a single word overlaps between the two headlines.

The threshold itself is the part worth tuning carefully. Set it too low and you start merging genuinely distinct stories that happen to share a topic — two different papers about "efficient attention," say. Set it too high and near-duplicates slip through again. I landed on checking similarity against both the title embedding and a short summary embedding, requiring agreement on both before calling something a duplicate. A single signal is too easy to fool in either direction.

The window matters as much as the metric

Comparing against every historical item you've ever stored doesn't scale, and it's also unnecessary — a duplicate detector's job is to catch things published within the same news cycle, not to notice that a story rhymes with something from six months ago. A rolling window (recent items only) keeps the comparison set small and keeps the semantics of "duplicate" tied to what it should mean: the same event, reported close together in time.

What I'd still change

At real volume, comparing every new item against every item in the window is the part that doesn't scale — it's an O(n) scan for every incoming article. The honest fix is a proper approximate-nearest-neighbor index instead of a linear scan, so dedup checks stay fast as the window grows. I haven't needed it yet, but I know exactly where the ceiling is.