Lens Distill
He started Venture Deals three times. He finished a pipeline. The book is still unread in order. The graph has 318 edges. This is the most Iresh project in the tree: a format problem treated as an infrastructure problem, then written up so the silent failures cannot be rebranded as features.
Lens Distill takes a PDF and a free-text topic lens and turns them into atomic claims with paragraph citations, a canonical concept vocabulary, and a sparse graph of prerequisite, related, and confusable edges. The Next.js app is a window. The work is a linear seven-stage job queue on Neon Postgres with pgvector. Parse is the boss fight. Silent fallbacks are how demos lie.
- distill.iresharma.com — live demo
- github.com/iresharma/lens-distill
- WatchIreshStruggle — I never finished Venture Deals, so I built a pipeline
Seven stages, one line
The whole system is a linear job queue. No DAG, no fan out, no orchestration framework, no cron. Each stage writes its rows and enqueues exactly one next job, or returns null and the book is done.
- 0 parse — PDF bytes to paragraphs with global indices. pdfjs-dist. Binary discarded; paragraphs stay.
- 1 chunk — ~1200 token windows, 150 overlap, hard break on chapter. js-tiktoken. Average-size gate catches exploded chapter detection.
- 2 embed — text-embedding-3-small, 1536-d, batch 100, HNSW cosine on Neon.
- 3 extract — Haiku, 40 chunks per job, concurrency 4, forced emit_claims tool. Persona is fenced user text, not the system prompt.
- 4 dedupe — cosine 0.86 clusters, 0.92 auto-merge, Sonnet only on the band in between. ~70% of wall clock.
- 5 canonicalize — string first, embeddings last. Negation prefixes never merge. participating vs nonparticipating is a regex, not a better model.
- 6 concepts — a tag becomes a node at ≥8 claims. primary_chapter is the modal chapter, not MIN().
- 7 concept_graph — Opus, typed edges, ≥80% coverage or throw. Prerequisite cycles broken with a recursive CTE.
Citations are a range check
Every paragraph is labeled [p412] in the prompt, using a global monotonic para_index as primary key (book_id, para_index). The model cites by marker. The code checks that every support_paras index falls inside the source chunk. Hallucinated [p9001] is dropped and counted. No LLM judge. Per-chapter indices were tried first; they collide on joins and the join succeeds.
Inline markers turn citation into a range check. Labeling paragraphs [p412] costs a few tokens and buys deterministic verification. Never let a model invent identifiers you plan to join on.
Fail loud, or the UI will look fine
A wrapped heading produced 157 chapters instead of 21. Nothing threw. Chunks fragmented, claims went context-free, ten dollars later the graph was a cloud of junk. The fix is gates: chapter count in [5, 40], demote 'chapters' with fewer than 15 paragraphs, drop headers that appear on >30% of pages, fail closed on page numbers unless offset agreement clears 0.8. The graph bug that shipped was worse: max_tokens truncated JSON, the parser returned [], the fallback drew a tag-only graph of disconnected dots, the book was marked ready.
Silent fallbacks are how demos lie. A fallback that produces a structurally different result (empty edges, fake pages, a tag only graph) is worse than a crash, because a crash gets fixed and a fallback gets shipped.
On Venture Deals: 3,001 paragraphs, 21 chapters, 109 chunks, ~978 live claims, ~54 concepts, ~318 edges. About an hour. About two dollars. Site-wide quota of three books per rolling week, advisory lock, PDF never stored. Forced tools, citation range checks, and a spend cap — in that order — because a deny list is the weakest layer and the one everyone reaches for first.
Portfolio demo: upload a PDF and an extract.md persona, run a real book-distillation pipeline, and inspect claims, concepts, a concept graph, and claim-embedding clusters. Drain starts automatically after upload (after() + chained continue). No cron worker, no manual stage-advance UI.