WORK / TECHPLUSE
TechPluse
An end-to-end pipeline that turns raw internet noise into a structured AI newsroom — discovered, deduplicated, summarized, and stored without a human in the loop.
ROLE
Sole builder
STACK
Python · FastAPI · Brave MCP · LangChain · PostgreSQL · React
TIMELINE
2025 — ongoing
STATUS
Live
THE PROBLEM
AI/ML news is scattered across hundreds of sources and most of it is noise — reposts, marketing, and duplicate coverage of the same paper from five different outlets. Following it manually doesn't scale, and generic news aggregators don't understand what counts as signal in this specific space.
I wanted a system that reads the internet so I don't have to, and hands back a structured, deduplicated, summarized feed — updated daily, without a manual review step.
CONSTRAINTS
- —Solo build, ran alongside a full-time job — no room for a heavyweight ops setup
- —No budget for a dedicated search index; had to lean on an existing discovery API rather than building a crawler
- —Needed same-day coverage: a 24-hour freshness window, not a weekly digest
- —Everything had to run unattended — no manual review step in the loop
ARCHITECTURE
A six-stage pipeline: discovery, filtering, dedup, rewrite, storage, and a scheduled trigger. Nothing in the chain waits on a human.
- 01Discovery — Brave MCP search across AI/ML/LLM queries, run every 24 hours
- 02Relevance filter — drops off-topic and low-signal results before they cost an LLM call
- 03Dedup — checks new items against recent entries for semantic overlap, not just title matching
- 04Rewrite — an LLM generates a clean headline and a structured summary per item
- 05Storage — PostgreSQL, with source links and direct PDF access preserved for research papers
- 06Schedule — the whole chain runs on a timer; nothing depends on me showing up
DECISIONS
1. A search API over a custom crawler
A hand-rolled scraper means maintaining selectors for every source and getting blocked constantly. Brave's MCP integration gave structured search results without owning that maintenance burden — the tradeoff is less control over source coverage, which is an acceptable cost for a first version.
2. Dedup on meaning, not on title text
Title matching alone lets near-duplicate coverage of the same story through, since different outlets phrase headlines differently. Comparing semantic similarity against recent entries catches those without needing an exact string match.
3. Fully automated, no manual queue
A review step would have been the easy way to keep quality high, but it also means the pipeline stops being autonomous — someone has to show up every day. Instead the filtering and dedup logic has to be trustworthy on its own, which is a harder bar to hit but the only one that actually scales.
RESULTS
24/7
Runs unattended
0
Manual review steps
Daily
Fresh coverage window
WHAT I'D DO DIFFERENTLY
The dedup logic today compares against a rolling window of recent entries; at real scale that comparison gets expensive. I'd move to a proper vector index for the dedup step from day one instead of retrofitting it later.