Distributed RAG Crawler
A distributed crawling and indexing system built to be operated, not just demoed. Scraping, link discovery, and indexing run as three separate BullMQ queues so each can scale on its own bottleneck discovery is cheap and wide, rendering is expensive and narrow. Six reliability controls keep it from being the kind of crawler that gets a domain blocked: robots.txt enforcement, per-domain rate limiting, retry with backoff, dead-letter queues for poison jobs, content hashing to skip unchanged pages, and page versioning so history isn't lost on re-crawl. On the retrieval side, PostgreSQL full-text search and pgvector HNSW embeddings run in parallel and are fused with Reciprocal Rank Fusion, which beats either strategy alone on queries that mix exact terms with fuzzy intent. The question-answering API returns grounded answers with citations over SSE.
Node.js · Fastify · Next.js · BullMQ · Redis · PostgreSQL · pgvector · Cheerio · Playwright · OpenAI
Three independently scalable BullMQ queues split scraping, discovery, and indexing so each scales against its own bottleneck
Six crawler reliability controls: robots.txt enforcement, per-domain rate limiting, retry/backoff, dead-letter queues, content hashing, and page versioning
Content hashing skips re-indexing unchanged pages; page versioning preserves history across re-crawls
Hybrid retrieval fuses PostgreSQL full-text search with pgvector HNSW embeddings through Reciprocal Rank Fusion
Cheerio for static pages, Playwright only where JS rendering is actually required keeps the expensive path narrow
Heading-aware chunking and table extraction preserve document structure that naive fixed-size chunking destroys
Grounded question-answering API returns citations with every answer and streams over SSE